A CI provider says it wiped and rebuilt thousands of Linux and Mac build machines after a security incident, then rotated credentials. Why take such a broad approach, and what should CI users consider?
Why wipe thousands of CI machines after a security incident?
Answers
Once a machine may have been compromised, proving that a cleanup caught everything can be harder than rebuilding it. The investigation may have a good idea of what was touched without being complete in the first few days.
The same logic applies to credentials. If you can't confidently rule out exposure, revoking a wider set avoids betting on an incomplete picture. That can be disruptive, though, especially if nobody has a reliable map of which services use each key.
Reply to this answer
Being able to rebuild at that scale is a separate achievement from deciding to do it. Managed infrastructure and provisioning automation make a huge difference when you need to reinstall services quickly. Bare metal Macs also bring a different set of operational challenges from cloud resources.
The provider said services didn't simply come back automatically. Their owners reviewed logs and got security approval first. That's a useful distinction between rebuilding machines and deciding when each service is ready to return.
Reply to this answer
If you use the provider, check its advisory and the recommendations for your own setup. CI jobs often have access to valuable credentials and automation, so work out which secrets and integrations your builds use and assess whether they need to be revoked or rotated.
Don't assume a provider rebuilding its machines automatically replaces every credential you manage yourself.
Reply to this answer
A mass wipe can look like a sign that an environment was poorly understood, but it isn't proof of that either way. It could also be a deliberate choice to contain uncertainty quickly. The useful questions are what was exposed, how the provider rebuilt safely, and what changes will make recovery easier next time.
Reply to this answer
Detection deserves scrutiny too. An alert for an unusual action might sound good in a preliminary response, but a rule that produces too many alerts across customers and employees can be hard to use. The details of how it's scoped and investigated matter.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Ravzelo.3m:
So the broad wipe doesn't necessarily mean they had no idea what was affected?
7a~fablevisor_362:
Right. It can also mean the remaining uncertainty wasn't worth trusting the old machines and credentials.