How do you manage configuration drift in your environments?

If you're using Kubernetes, the best strategy is to implement strict GitOps with Argo or Flux and enable autosync for anything in the cluster. It's crucial that only the ops team has direct access to the cluster to minimize drift. For broader infrastructure with multiple components, although Terraform is theoretically a great solution, in practice it’s tough to keep everything aligned. Small changes, like shifts in provider APIs, often lead to drift.

We run a daily Ansible job that helps align everything efficiently back to our desired states. It's a straightforward approach that keeps our configurations under control.

The key for us is having a centralized automated pipeline to deploy any changes triggered by a Git push. No one gets admin access unless it's an emergency, which keeps drift at bay. Plus, we have daily GitHub Actions that run with a Terraform plan to monitor for any unintended changes, helping us catch any issues early.

We primarily use source control for changes, which simplifies management since I'm often the only one making updates. It also helps to manage tools like Puppet in source control to keep everything organized.

Some folks argue that Kubernetes is overkill for most companies, but it really addresses these types of issues effectively. Tools like Argo or Flux can keep everything synchronized, automatically reverting any changes that aren't approved. For example, if a developer adjusts a config map, Argo can revert it back within 30 seconds! There are other tools like Chef, Ansible, or Salt, but those require you to run them on a schedule to keep things in sync.