Metrics are critical to conveying the value of your software security initiative (SSI) to senior management, who need this concise input to inform their investment decisions. However, like business objectives themselves, those metrics vary widely from business to business and can vary greatly over time (e.g., when the new CEO shows up or when work-from-home happens).
The security-related questions that senior management asks often include cultural issues (are my people happy?), business issues (am I helping productivity?), and technical issues (are my technology stacks working?). We also live in an environment that is increasingly automated for continuous security, and one where DevOps and security teams are more frequently working together—an emerging DevSecOps culture and capability. What’s the impact on how organizations approach metrics? There’s no question that the DevSecOps movement is changing the way organizations collect reporting data. But how has it changed? And what are the KPIs that are valuable to track today?
“Many are inheriting AppSec responsibility for the first time,” says Sammy Migues, principal scientist with Synopsys and one of the authors of the BSIMM—the Building Security In Maturity Model. “Some are running into the DevOps efforts going on in engineering, and they’re struggling with how to get security over into DevOps as well as how to understand the results of and manage that effort in their metrics programs.”
Migues notes that there is no one specific way to choose and visualize AppSec metrics. However, there are some factors to consider when collecting measurements and crafting metrics that can help convey progress and value within a specific SSI. Software security managers can reflect on some of the following questions when it comes to measuring success and value in SSIs.
How are we doing with embedding Dev into DX?
Some research finds as many as 70% of organizations are undergoing digital transformation (DX) initiatives. What kinds of metrics are meaningful when it comes to DevSecOps and DX? Migues suggests tracking factors that provide clarity into DX progress. For example:
- How far along is the organization in implementing DevOps across the engineering teams and the software portfolio?
- How much infrastructure as code has been put in place to replace error-prone human processes?
- How effectively are DevOps teams implementing sensors in toolchains and production to detect conditions requiring intervention?
- How much success has the business had with replacing people with bots for repeatable processes?
“Think through ‘What are the highly manual things today?’ and then say, ‘Can we automate those things?’”
For example, says Migues, secure SDLC gates have historically been highly manual and present an opportunity for change. “In a digital transformation journey, how are you doing at turning those highly manual, people-driven things into code?” he says. “What are the big, people-heavy things that introduce friction to development and deployment, and what is your metric from showing progress moving from a bad place to a good place? How quickly am I removing people from the process, for example?”
How is our champions program helping to remove friction?
Champions programs are another place where organizations can measure success and progress, says Migues.
“Today we need to measure champions’ impact on security as well as on digital transformation and friction removal.”
Migues says that to advocate effectively for more efficient processes, champions need to add to their digital toolset and their security knowledge as part of an organization’s digital transformation process.
“The old metric was something like having 50 developer teams and 10–20 champions covering those waterfall (or agile-fall) teams. Now those teams are CI/CD and DevOps. Simply having a person to call isn’t fast enough. How is that champion injecting their security knowledge into those teams, and what is the coverage of that effort? Are they translating governance requirements into code and sensors that apply to all the development streams?”
Migues suggests some metrics to measure champion effectiveness might evaluate how they’re helping facilitate infrastructure as code securely, or how they’re facilitating container security and cloud security initiatives. Of course, the basics of value stream, CI/CD toolchain, and DevOps group coverage also remain important.
“This allows the devs to focus on feature velocity while champions are taking their security knowledge and translating it into operational verification of adherence to expectations. That’s great value and scales well.”
Which test methodology can we use as a function of risk?
In an environment that includes red teaming, penetration testing, and static analysis, which one should be applied, and when, as a function of risk?
“What is a metric of risk that determines if I have appropriate coverage?” says Migues “You need different testing methodologies at different times, to run at different speeds. And you need them to accomplish risk management collectively. As a foundational metric, can you apply the appropriate testing, at speed and scale, as soon as the related artifact comes into existence? Will your pipeline do SAST as soon as there is code in the repository? Will it do DAST as soon as there is buildable running code? Will it do configuration reviews as soon as it knows about containers and other artifacts? Are the defects provided to engineers in cadence with their processes and in the tools they already use?”
It’s important to set your own thresholds for coverage, speed, friction, contribution to resilience, and other important objectives and then use metrics such as these to see whether you’re making the expected progress.
How is our coverage with governance as code?
As organizations look to increase automation and place more workloads in the cloud, governance as code allows them to codify how applications and infrastructure run. It means moving away from a manual, human-based approach to an automate-first mentality to using that approach to enforce internal policies and best practices.
Governance as code increases agility and efficiency as CISOs can define and automate these best practices. As Migues points out, it means moving cumbersome, low-tech processes of enforcement out of the way in favor of automated, efficient processes.
“It’s taking a corporate AppSec program, pulling it out of wikis, and embedding it in the DevOps process,” says Migues. “Ask: How am I going from governance as PowerPoint to governance as code? How am I facilitating feature velocity while also accomplishing the appropriate level of risk management?”
What is your approach to gathering and reporting metrics? Please share your experiences so we can all learn together.