I'm trying to understand how teams handle conflicting infrastructure information in real environments. For example, Terraform indicates that an EC2 instance should have encryption enabled, while the AWS console or API reports that encryption is disabled, and the CMDB still marks the instance as compliant. How do you determine which system reflects reality? Do you establish a formal source of truth, or investigate each discrepancy manually? What methods do you use to distinguish actual configuration drift from stale data or a deployment that only partially succeeded?
3 Answers
The live cloud API is the authority for what exists and is configured right now. Terraform state reflects what Terraform believes it last managed, while the configuration files describe the intended state. I’d run a plan or refresh against the real account, inspect the apply history and logs, and use audit events to see whether someone changed the resource outside Terraform. The CMDB should generally be treated as a downstream report rather than proof of current compliance.
Long term, the fix is governance as much as tooling. Use periodic plans, restrict production changes with RBAC, and require emergency console changes to be reconciled back into code. If people can routinely modify infrastructure manually, no system will remain a dependable source of truth. Terraform code can be the intended authority, but only if the organization actually enforces that workflow.
Exactly—the code should define the desired state, but it cannot override what is actually running. If the organization allows unmanaged console changes, the discrepancy is a process problem rather than something Terraform alone can solve.
There are really three different sources involved: the cloud API tells you the actual state, Terraform configuration expresses the desired state, and Terraform state records Terraform’s last known relationship with the resource. A mismatch between the API and configuration is drift or an intentional change that was never codified. A mismatch between the API and state can point to an incomplete apply, stale state, or an out-of-band change. I’d verify the API first, then compare it with a refreshed Terraform plan and deployment logs.
A refresh-only plan is useful for detecting changes to resources Terraform already tracks, but it won’t reliably identify resources created outside Terraform. Audit logs are important for catching those cases.