I'm dealing with some challenges related to Infrastructure as Code (IaC) drift because we use AWS Config and Security Hub for auto-remediation. For instance, tasks like enabling default encryption for S3 buckets or adjusting security group rules work well automatically, but they create issues. The problem arises as Terraform doesn't recognize these changes, leading to situations where the next plan either tries to revert the adjustments or I end up having to manually edit the state. I'm curious about how other teams tackle this issue. Do you simply accept the drift and manually update Terraform? Do you bypass auto-remediation and manage compliance through your standard IaC processes? Or do you have a different approach? I've had discussions suggesting that auto-remediation clashes with IaC principles, advocating for workflows where compliance findings are addressed directly in Terraform through pull requests. Does anyone resonate with that or have different experiences?
6 Answers
At the end of the day, auto-remediation can lead to confusion. It's similar to someone logging into the console and making changes out of the blue. Terraform isn't automatically aware of those changes. You either need to import those new modifications or use the auto-remediation as a clue for what needs adjusting in your code.
Auto-remediation of resources managed by Terraform is a recipe for headaches. It's hard because it can lead to confusion and unexpected changes that Terraform isn't aware of. It's like a setup for disaster in the long run!
A good strategy we've employed is to split remediation into two tiers. For serious risks, like exposed S3 buckets, we keep auto-remediation in place but have it trigger a pipeline to open a PR for Terraform updates. For less critical issues, we just monitor and alert, leaving those changes to the IaC pipeline. Most compliance findings don't need an urgent fix and can wait for proper updates.
Implementing Terraform config rules in your pipeline using tools like Trivy or Rego policies can prevent incorrect deployments from happening in the first place. If a manual change happens, that's when you can focus on remediation.
Consider connecting EventBridge to AWS Config. Each time remediation is triggered, it could create a Jira issue for better tracking and handling of changes, so nothing slips through the cracks.
You're spot on! The idea that auto-remediation conflicts with IaC is true. Your Terraform should be the source of truth. Changes made manually or automatically can cause major issues down the road.
Yeah, I honestly wish our systems opened issues or PRs automatically! Instead, we just get a mountain of scan data to sift through while our security measures often block changes that would trigger auto-remediation.
I completely agree! Prevention beats correction where possible. But AWS is tricky, especially with automatically created resources like tags. Those can’t be controlled easily with SCPs, making things complicated when trying to add them to Terraform.