The Fault in Our Metrics: Rethinking How We Measure Detection & Response | A Conversation with Allyn Stott

Podcast Redefining CyberSecurity with Sean Martin

In this episode of The Redefining CyberSecurity Podcast, host Sean Martin converses with Allyn Stott, who shares his insights on rethinking how we measure detection and response in cybersecurity. The episode explores the nuances of cybersecurity metrics, emphasizing that it’s not just about having metrics, but having the right metrics that truly reflect the effectiveness and efficiency of a security program.

Stott discusses his journey from red team operations to blue team roles, where he has focused on detection and response. His dual perspective provides a nuanced understanding of both offensive and defensive security strategies. Stott highlights a common issue in cybersecurity: the misalignment of metrics with organizational goals. He points out that many teams inherit metrics that may not accurately reflect their current state or objectives. Instead, metrics should be strategically chosen to guide decision-making and improve security posture. One of his key messages is the importance of understanding what specific metrics are meant to convey and ensuring they are directly actionable.

In his framework, aptly named SAVER (Streamlined, Awareness, Vigilance, Exploration, Readiness), Stott outlines a holistic approach to security metrics. Streamlined focuses on operational efficiencies achieved through better tools and processes. Awareness pertains to the dissemination of threat intelligence and ensuring that the most critical information is shared across the organization. Vigilance involves preparing for and understanding top threats through informed threat hunting. Exploration encourages the proactive discovery of vulnerabilities and security gaps through threat hunts and incident analysis. Finally, Readiness measures the preparedness and efficacy of incident response plans, emphasizing the coverage and completeness of playbooks over mere response times.

Martin and Stott also discuss the challenge of metrics in smaller organizations, where resources may be limited. Stott suggests that simplicity can be powerful, advocating for a focus on key risks and leveraging publicly available threat intelligence. His advice to smaller teams is to prioritize understanding the most significant threats and tailoring responses accordingly.

The conversation underscores a critical point: metrics should not just quantify performance but also drive strategic improvements. By asking the right questions and focusing on actionable insights, cybersecurity teams can better align their efforts with their organization’s broader goals.

For those interested in further insights, Stott mentions his upcoming talks at B-Sides Las Vegas and Blue Team Con in Chicago, where he will expand on these concepts and share more about his Threat Detection and Response Maturity Model.

In conclusion, this episode serves as a valuable guide for cybersecurity professionals looking to refine their approach to metrics, making them more meaningful and aligned with their organization’s strategic objectives.