TJ-Actions Breach: How One GitHub Action Exposed Thousands

In March the cyber security world was rocked by the compromise of TJ-Actions. 23,000 users were put at risk as hackers took control of TJ Action’s code and used it to leak the secrets of all its users.

A few weeks later the dust has finally settled, and we can finally deep dive into what happened and how we can stop it in the future.

What is TJ-Actions?

TJ-Actions is a collection of open source GitHub Actions that help you build out fast and efficient CI/CD pipelines. These are incredibly useful if you are an engineer looking to build CI/CD pipelines as a lot of the hard work is then done for you by TJ-Actions.

TJ-Actions are incredibly popular and used by 23,000 users and is provided for free by a small group of volunteers.

GH-Actions Logo

It would tell you which parts of your code base have been changed by each pull request/commit. This means you can trigger certain CI/CD pipelines based only on the files changed instead of having to run all your pipeline all the time.

The Breach

At 4pm on March 14th the alarm was sounded about TJ-Actions being compromised.

The attacker had managed to steal a Personal Access Token (PAT) from a bot that managed the TJ-Actions repos. This gave them write access to the repository containing the Changed-Files code. From here they were able to create a malicious commit containing a script that:

Downloaded and ran a python script from a gist file
This script then scanned memory for secrets and then printed them to public logs
The attacker then updated every previous version of TJ-Actions to point to this malicious commit. This means if you were using an older version of TJ-Actions, you were still redirected to this new malicious version and would start printing all your logs.

Stealth Mode

The attacker also tried to hide their activity using a few different techniques:

They double base64 encoded the script to make it harder to detect
They changed the user in the commit message to make it look like changes were coming from someone else.
They pushed the tag changes remotely, this hides the changes from the public audit logs and makes it harder to see what version changes had happened.

However, for a while the main question was how did they get access in the first place?

Initial Access

The method they used to gain access was surprisingly ingenious and highlights the complexity of modern supply chain attacks.

TJ-Actions had done everything right, their expected security controls were all in place. The PAT token they used to compromise TJ-Actions came from a contributor who also contributed to other repositories including Review Dogs.

Review Dogs was compromised by an attack on one of its dependencies, an open source project called SpotBugs.

This means the attacker moved through two different open source repos to get access to TJ-Actions.

Within Spot Bugs a maintainer had used their PAT token in a GitHub Action to debug an issue. The attacker managed to get access to this PAT token by:

Requesting and receiving write access as a maintainer of Spot Bugs
Forking the repository containing the PAT token access
Malicious editing of a GH action in the forked repo that had a pull_request_target trigger which provided access to the PAT token

The pull_request_target trigger allows you to trigger a job on pull requests even when they are in forked repos, and give them access to your repository’s secrets.

on:
    pull_request_target:
        types:
            - opened
            - synchronize
jobs:
    check_branch:
        runs-on: ubuntu-latest
        steps:
            - name: Check branch name
              if: startsWith(github.head_ref, 'feature/')
              run: echo "This pull request is from a branch starting with 'feature/'"
            - name: Notify
              if: !startsWith(github.head_ref, 'feature/')
              run: echo "This pull request is not from a 'feature/' branch"

Understanding the TJ-Actions breach: A cautionary tale for securing your CI/CD pipelines.

This meant that the attacker could start making changes in the forked repo without anyone in the main repository realising their secrets were being used in malicious jobs.

The second he had access they created new accounts for themself which they then used to compromise the main repository and start leaking secrets from any projects that used Spot Bugs.

Since Spot Bugs is used by Review Dog they managed to move laterally into Review Dog and leak the PAT token of one of the maintainers. This was then used to compromise the repos within TJ-Actions.

The Impact

The main goal of supply chain attacks like these is to typically to “move laterally” onto the dev machines of engineers working on the products. From there they can work out what other opportunities they might get.

It’s not known if this was an opportunistic attack or if the attacker was deliberately targeting someone. However, once they got access to TJ-Actions they tried to release new malicious payloads specifically targeting Coin Base, one of their customers. Thankfully their attack against Coinbase was not successful.

Others weren’t so lucky, analysis has shown that around 5500 repos referenced the malicious GH actions and would have been impacted. From these at least 220 are known to have printed their secrets to their public logs.

These secrets included:

GitHub install access tokens (which give you access to the GitHub repo but expire in 24 hours)
DockerHub access tokens (could allow you to overwrite docker containers)
NPM repo access tokens (could allow you to compromise NPM packages)
AWS tokens (could give you access to AWS tenants)

All of these are serious compromises that on their own could have made the news. If the attacker managed to access these secrets and use them, before they are changed, then we can expect more supply chain attacks to appear soon.

What to do if you were compromised?

The malicious code has been deleted from the GitHub Gist. This means that the risk of now being exploited is very low (unless you have cached the local code).

In the future if you think are compromised by a malicious GitHub action, the traditional incident response should be to:

Audit your GitHub logs for suspicious IP addresses or secrets being leaked.
Rotate all active secrets being used in your repositories.
Identify what malicious external GitHub Actions you are using and remove them. completely until they are considered safe to use again.

How can we defend better next time?

If you are a maintainer of open source there are a few actions you can take to reduce the risk of being compromised.

Validate all contributors you give write access to your repos.
Make sure not to use secrets like PAT tokens in your repos.
Make sure your secrets are not leaked publicly and rotate them if they are.
Avoid using pull_request_target in your GH Actions.

Validate all pull requests submitted to make sure don’t contain malicious code before approving.

If you are using open source and want to make sure you stay protected from attacks like these, you can make sure to:

Never tag to the latest version of a GH action
- This means if an attacker compromises the action and releases a new malicious version you will instantly start using it
Only tag to a specific commit hash, NOT a version
- In this compromise the attacker redirected all past tags to a new malicious version. This mean if you were using a specific tag you would still be compromised. If you tag to a commit hash instead this will prevent the attacker from being able to redirect you to a malicious release.
Follow the official GitHub guidance found here.

The Open Source Reality

One important thing to remember is that this isn’t the fault of the maintainers of any of the open source projects. The blame lies solely with the attacker. Even if you do everything right there is still a risk of being compromised, it’s all about managing that risk and making sure you are doing the core things you can to keep yourself protected.

We can help prevent attacks like these by making sure our open source communities have the resources to protect themselves. The best way to do that is to contribute to open source repositories that you use. Your contributions will help them be able to handle security issues more quickly and efficiently. This is one of the best ways to mitigate against serious risks you might face from open source packages. TJ-Actions accepts donations through the open collective.

If you are open source and want to make sure contributions from the community are safe. We can offer scanning of Pull Requests to make sure they are safe to use.

Conclusion

The incident around TJ-Actions is not the fault of the TJ-Actions community. Attacks like these are on the rise and organisations that are doing everything right continue to get compromised.

We need to learn how to adapt and improve from these situations and build our better Software Supply Chain and Vulnerability Management programs that can adapt to the change in the threat landscape. This is what OSSPREY aims to do and how we can help protect you from the latest and most advanced software supply chain attacks.

How Ossprey Can Help

Ossprey uses our AI platform to scan and identify malicious open source code before it can compromise you. By integrating Ossprey into your platform you can depend on any open source code without having to worry about compromise knowing that we will detect and warn you before you use anything that could be malicious.

Learn more about OSSPREY and how we can help you here