Spring4Shell puts lessons learned from Log4j to the test
Yet another critical open source vulnerability has been found in common open source software even as some security teams are still reeling from the last big issue.
Last December, a vulnerability in Apache Log4j—the widely used Java logging library—caused big problems for application security teams. The countless disruptions have been remedied for Log4j 2, but vulnerable versions were downloaded more than 10 million times. Almost every major Java-based enterprise software and server on the market use Log4j—including cloud platforms, online applications, ecommerce tools, and email services. (Read Tales from the Trenches: Log4j Incident Response.)
The Log4j vulnerability could let hackers conduct remote code execution (RCE) attacks on target systems. The vulnerability first showed up in the Minecraft video game, as well as in enterprise applications and products from Apple, Microsoft, Amazon, VMWare, Cisco, Cloudflare, and Twitter. At one point, Checkpoint Research reported 100 attacks per minute related to Log4j, and more than 4.3 million attempts were prevented from granting access to the vulnerability.
This was a major a wake-up call that appsec teams might not know their open source software and components as well as they thought. To prevent malicious intrusions in the future, development teams need to know what code is bundled into the software that runs their businesses. “It distills down to knowing what’s in your software, whether it’s software you purchase or software you build, and knowing what’s in the open source dependencies that you build on top of. Everything else can be derived from that,” said Mike McGuire, senior product marketing manager at Synopsys.
Last month, two new vulnerabilities affecting different Spring projects were identified that could allow attackers to run arbitrary code on vulnerable systems. Widely referred to as Spring4Shell, this pair of vulnerabilities is yet another high-severity issue in open source software for appsec teams.
DevOps teams have to deal with a key shortcoming in the way they manage open source software, according to McGuire. “Organizations found that they didn’t know what powers their applications. Software purchased from a vendor doesn’t always come with a complete inventory of the components, but it’s vital to know what’s in your applications. We’ve found that organizations often have twice as much open source as they think they have.”
Software composition analysis tools
According to McGuire, some of the initial pain associated with the Log4j vulnerability came from appsec teams needing to sift through pages of open source application code just to find out if they were attacked. For this task, software composition analysis (SCA) tools is key. DevOps staff using an SCA tool that was already in place and operating had an easier time remediating intrusions than those that didn’t.
“In those organizations with an SCA tool, managers woke up to an alert via email or Slack thread saying there’s a vulnerability that affects their organization. They immediately knew which of their applications were affected, which versions of those applications were affected, and got valuable input on how to go through and remediate this issue and stopgap it as quickly as possible. They probably could have done this by 10 a.m. on the day of the disclosure, and the rest of the day, they could do a victory lap,” McGuire said.
But the process wasn’t so easy for teams without an SCA tool. These managers may have woken up and found out about the vulnerability by reading headlines or, in the worst case, via an email from the CEO or CISO.
“They had to dig through their applications to find which ones, and then which versions, contained Log4j, “ McGuire said. “Think for example about how many versions of iOS are out there and how many different people are running different versions of iOS. That’s many, many layers deep, and they had their bosses breathing down their necks. When they did finally figure all that out, they had to figure out how to remediate it. This is stuff that they might not have in-house expertise to do. Maybe they have a security team, but the security team isn’t up to snuff for this particular component.”
Software Bill of Materials
In addition to SCA tools, the software Bill of Materials (SBOM) has emerged as the most important security tool for organizations addressing Log4j.
An SBOM is an inventory of open source and third-party software components inside applications. It contains licenses, versions of components, and patch status, all of which allow security teams to quickly identify associated security and license risks. Because open source is growing in popularity, this inventory has become the most important part of software security management. However, for it to be most effective, development teams must keep the SBOM up-to-date.
“All roads lead to the SBOM, and it drives all your efforts going onward,” McGuire said. “It’s at the center of an open source risk management program. The question becomes how to leverage information in it, which is a single snapshot in time. If you have a six-month-old SBOM that you’re still depending on, it’s better than nothing, but it’s tough to rely on that. What if six months ago you weren’t using Log4j, or you were using a different version and it was upgraded to the vulnerable version, and you don’t know that because you have this snapshot from six months ago? To be most effective, a software Bill of Materials must be compiled on a very regular basis. I’m talking every build,” said McGuire.
This is especially crucial because software development organizations are building and releasing applications two or three times per day, not once or twice per year. "The SBOM lets organizations check their open source components against known vulnerabilities. With it organizations can become proactive and avoid bringing new vulnerabilities into their applications,” said McGuire.
“You want to bring it to developers so you can catch these issues and avoid them or fix them before they become more expensive. You want to create a gate for components that can be brought in. A lot of organizations actually have a vetting process, so if a development team or a software architect wants to leverage an open source project or component, they’ll go through a process with a security team that will review it, and say yes or no to it,” McGuire said. “From there, developers are welcome to use it. It’ll get scanned and added to an SBOM, it can be upgraded to different versions or downgraded if need be, and those versions will be scanned for vulnerabilities.”
Application security teams also need to ensure that the preparatory work involved with documenting and scanning for vulnerable libraries is a regular process, not an incident-response task. In other words, as business-as-usual, not as a one-time directive.
“This is an absolutely nonoptional part of the software development life cycle, and it needs to be built into every process. You need to build security into it,” McGuire stressed. “It’s about planning ahead of time and putting the processes into place so that when this does happen, you have a defined process. That’s why you need solutions and tools that are built into your existing tool chain.”
It’s also necessary to have a mindset that goes beyond tools. McGuire says the basic three pillars are people, processes, and technology, all of which need to be defined.
“Every organization is going to have its own unique risk tolerance. A financial services institution is going to have a very different risk tolerance than an ecommerce platform, or an organization that’s building an application that’s never going to be hosted on a public cloud, for example. You need to make sure you define the processes for what best fits you. You need to make sure the tooling fits into your tool chain at the right time, and you need the people to make sure they’re leveraging the tools so that they’re getting the most out of them. Make sure they’re not counterproductive, not slowing product delivery, and they fit your unique risk tolerance and unique workflows as well. This will make it more about risk management rather than incident response,” McGuire said.
In the end, it’s the software Bill of Materials that will keep organizations from having to put out fires when the next vulnerability gets disclosed.
“An accurate software Bill of Materials will let you know what’s in your software so you can act. That’s now actionable, and it all boils down to that. Know what is in the software that powers your business and know with confidence,” concluded McGuire.
Application security teams that spent the time building SBOMs, deploying SCA tooling, and building open source risk management processes are now putting those efforts to the test by responding to the Spring4Shell vulnerability. Teams that didn’t are faced with the same uncertainty and painful incident response they had to deal with in December.