I recently encountered a concerning issue with my AWS keys possibly being exposed. AWS flagged some suspicious activity where my account made a GetCallerIdentity call from a French IP address. A review of CloudTrail showed numerous attempts to GetAccount and CreateUser using my credentials, which were supposed to be strictly for deploying static assets to S3 and invalidating CloudFront distributions (with limited permissions). Thankfully, it looks like those attempts were unsuccessful. After realizing this, I deleted the compromised keys, tightened my permissions, and switched my GitHub actions to use OIDC instead of AWS access keys. I'm puzzled about how my keys were leaked in the first place since they were only stored as secrets within GitHub Actions. To clarify, my repository is private and intended for a personal testing project. I also recently enabled 2FA in GitHub after realizing I didn't have it set up before.
6 Answers
Keys exposed in public repositories can be exploited almost instantly, which is part of why AWS and GitHub have systems to detect and respond quickly. Since you're unsure how they may have leaked, it’s best to start by reviewing the system that generated these keys. If they weren't exposed through a shared or public repository, there may be a chance your development system was compromised.
Just a heads up, if your workflow logs were accessible to the public at any point, those secrets could have been exposed. If you're running on shared GitHub runners, there's also a risk that someone could have extracted your keys from memory after your job executed. Keep your logs private to avoid that issue!
My repo is private, but I was using the free GitHub hosted action runners.
Using OIDC for authentication is a good step forward! This way, you can eliminate the need for AWS access keys, which are often vulnerable. Just make sure to follow the documentation carefully to set it up correctly.
Thanks! I’ve set it up and even added a canary secret in my GitHub secrets to monitor for potential compromises.
About a month ago, a popular GitHub Action got compromised, leading to a leak of CI/CD secrets. If your action isn’t well-audited or designed with security in mind, it’s worth checking if you might have used that one or something similar.
You’re heading in the right direction by switching to OIDC and tightening up your security. Just remember to audit those third-party actions you used. If they aren’t pinned to specific versions, it creates a security hole that might get exploited. Check your action logs, especially around the time when the suspicious activity started.
Are you using any third-party actions from sources other than GitHub or AWS? Actions that aren't pinned to specific versions can be manipulated over time, making them a potential risk for your CI/CD pipeline. It’s crucial to review any actions you've added to ensure they’re safe.
I totally get why you'd want to be cautious. Using unverified code in CI/CD can lead to huge security issues.
I should mention that my repo is private and just for personal testing. I've also changed my password and enabled 2FA since realizing I didn’t have it before.