CI/CD pipeline attacks: A growing threat to enterprise security

CI/CD pipeline attacks are a growing threat to enterprise security. In this article, we’ll provide an overview of CI/CD for non-developers, discuss the cybersecurity issues involved, and offer some recommendations for developers, companies, and security teams.

What is CI/CD?

CI/CD stands for two complimentary DevOps software development practices: continuous integration and continuous delivery (or, less commonly, continuous deployment).

In continuous integration, code changes made by different developers are compiled, tested, and merged into a single, shared code branch. The process is frequent and automated. 

Continuous delivery means that new software is checked and uploaded to a shared repository automatically. At this point, it’s essentially “ready to go”: Operations teams can see it and manually deploy it to end users at will. If CD refers to continuous deployment, as opposed to delivery, it just means that the new software is actually rolled out to users automatically.

The various steps and processes of CI and CD, along with the tools, platforms, and repositories that enable them, are known as the CI/CD pipeline. The point of CI/CD is to simplify, streamline, and automate large parts of the software development process—so updates get out to end users faster and more reliably.

It’s a huge benefit, which is why so many development teams have incorporated CI/CD into their workflows. But all of the automation, and the third-party tools, have a cost as well: namely, the risk of a serious compromise if bad actors manage to subvert any part of the CI/CD pipeline. And this is exactly what a growing number of cybersecurity professionals are warning about.

Why cybersecurity professionals are worried about CI/CD pipeline attacks

CI/CD is not new. But threats to the CI/CD pipeline are being discussed more and more in security circles of late. Why?

To begin with, the stakes are genuinely high. A successful CI/CD pipeline attack could result in a critical compromise. As one DFIR expert we interviewed put it: “If you can own the pipeline, it’s game over. I don’t know if I’d sleep very well if I was a security engineer for one of the major CI/CD SaaS products or vendors.”

But beyond the potential seriousness of pipeline exploits, there’s another factor at work as well: despite its growing popularity, CI/CD is still just getting started. Research conducted by the Continuous Delivery Foundation last year found that a full “60% of enterprise developers use CI/CD in their workflow,” and that even among developers for smaller companies, the CI/CD adoption rate approaches 50%. However, the same research showed that only 18% of all developers are using both CI and CD to fully automate software delivery. 

The meaning is clear: There is a huge and as-yet unrealized potential for CI/CD adoption—which will mean an ever-increasing number of targets for attackers in the future. And the attackers, as we have seen time and again, have a tendency to go where the targets are! The threat is on the horizon. The CI/CD pipeline may very well be the next major battleground for enterprise security teams.

The impact of CI/CD exploits

It’s obvious that you don’t want bad actors to gain access to CI/CD infrastructure used to build and deploy software. But in real-world terms, what could result from a CI/CD pipeline compromise?

To answer the question, consider what CI/CD is: a way of getting software to end users—software that those end users will trust.

In a worst-case scenario, a CI/CD compromise could be used to install malicious components in an application that is then pushed out to large numbers of users. Depending on the software, and on the types of organizations using it, this could be catastrophic. During the SolarWinds incident in 2020, for example, bad actors were able to install backdoors in the company’s Orion network monitoring software. That software was used by Fortune 500 companies, the U.S. government, and the military.

But beyond the rather dramatic examples of bad actors using a compromised CI/CD pipeline to poison software, there are other concerns as well. For example, the GoCD CI/CD vulnerability, discovered by security researchers in Switzerland last year, could have allowed an attacker to view a company’s sensitive information, leak their API keys, and steal their proprietary source code. And the more recent GitHub attack resulted in the actual theft of login details for 100,000 npm accounts, resulting in large-scale data exfiltration. 

This is why the threat to the CI/CD pipeline is being taken so seriously at the highest levels of government. In the most recent National Security Telecommunications Advisory Committee (NSTAC) Report to the President on Software Assurance, the CI/CD pipeline was cited as one of four “major security aspects” affecting the software supply chain.

How can the CI/CD pipeline be protected?

Securing the CI/CD pipeline is everyone’s responsibility. But depending on who you are, this will require different steps. Here are some general recommendations for developers, companies, and security teams:

Developers 

Developers need to move beyond DevOps and embrace a SecDevOps approach, incorporating security into every aspect and stage of the software development cycle. 

With respect to pipeline security specifically, the UK government’s National Cyber Security Centre (NCSC) has published a useful guide to CI/CD best practices for developers.

At a bare minimum, developers should secure CI/CD-related accounts with strong credentials and multi-factor authentication. In addition, they should take care to protect credentials and authentication tokens from inadvertent exposure, e.g., through misconfigured cloud storage, publicly accessible hardcoded credentials, and so on. This may seem like a basic recommendation, but security assessors routinely find such “obvious” lapses when auditing CI/CD pipelines for vulnerabilities. NCC Group, for example, has published a blog post entitled “10 real-world stories of how we’ve compromised CI/CD pipelines” that makes for instructive, if somewhat disconcerting, reading.  

It’s also important for developers to harden a pipeline against lateral movement or privilege escalation in the event of a successful attack. Devs should operate according to the principle of least privilege at all times. NCSC also recommends strong isolation to protect builds from one another. This way, in the event of a compromise, it’s far easier to contain the damage.

Companies

Businesses need to know that their software developers are performing their work according to industry-standard best practices. 

In terms of pipeline security, this is largely an issue of due diligence. If you have an in-house security team that can handle this, great. Otherwise, you may need to work with a third-party security consultancy to audit vendors and make sure they are following best practices for CI/CD pipeline security. 

That being said, because the modern enterprise software ecosystem is so vast, it’s not possible to guarantee that everyone is doing what they should be doing to secure the pipeline. For businesses, the scary thing about CI/CD pipeline attacks (and supply chain attacks generally) is that you can do everything right and still end up exposed. For this reason, it makes sense to adopt a defense-in-depth strategy that includes endpoint security, software-defined secure networking, data backup and disaster recovery precautions, and incident response planning.

Security Professionals

For security teams, helping developers and companies secure the CI/CD pipeline is a matter of careful enumeration…and of not taking anything for granted. You might not think that an experienced developer would leave hardcoded access credentials lying around in a public repository. Alas, in many cases, you’d be wrong! Be sure to cover all your bases.

In addition, it’s essential to gain better visibility into what is actually happening on CI/CD platforms. Log all sensitive system, repository, and user events, and keep track of authentication requests. Then audit your logs by running pre-defined detection and response (D&R) rules on the collected data. By automating the logging and auditing of CI/CD pipeline activity, you stand a better chance of spotting malicious activity when it happens—and of being able to take rapid action to mitigate an event if needed.

How LimaCharlie can help 

LimaCharlie offers on-demand, cloud-native cybersecurity primitives to security professionals. To address the growing risk to the CI/CD pipeline, we’ve introduced a GitHub sensor that allows GitHub audit logs to be ingested directly into LimaCharlie. You can monitor system, user, organization, and repository events—using custom D&R rules to automate alerting. To watch a live demo of GitHub access detection and auditing with the LimaCharlie GitHub sensor, check out our recent webinar.

For MSSPs looking to help their clients build a more robust cybersecurity posture and counter next-generation threats, LimaCharlie offers 100+ integrations and security capabilities, including EDR, software-defined networking, WEL monitoring, and more.