How do you handle conflicting infrastructure state in a real environment? For example, Terraform indicates that an EC2 instance should have encryption enabled, while the AWS console or API reports that encryption is disabled. Meanwhile, the CMDB still lists the instance as compliant.
How do you determine which system reflects reality? Do you define one authoritative source, or investigate each discrepancy manually? What methods do you use to distinguish genuine drift from stale data or a deployment that failed partway through?
3 Answers
A refresh-only plan is useful for detecting changes to resources Terraform already manages, including an encryption setting that was altered outside Terraform. It will not reliably identify resources that were created completely outside Terraform, though, so pair it with cloud audit events such as CloudTrail. That combination helps separate configuration drift from missing resource ownership or a deployment that never completed.
Ideally, the Terraform configuration in source control defines the desired state, while the provider API confirms the actual state. Those are different kinds of truth, so neither should be confused with the other. If people can change infrastructure directly through the console, discrepancies will keep appearing. Periodic plans, restricted permissions, audit logs, and a clear process for emergency changes help make Terraform the intended control point instead of merely documenting changes after the fact.
For the state of the running infrastructure, the live cloud API is the authority. Terraform state only records what Terraform believes it applied, and a CMDB is usually best treated as a downstream report rather than evidence. Run a Terraform plan against the actual account: if it detects a difference, that points to drift or an out-of-band change. If the plan is clean but the API still disagrees, investigate whether the apply failed, the state is stale, or the resource attribute is being interpreted incorrectly.
Exactly—Terraform code should be the desired source of truth, but it cannot override the fact that the cloud account is currently configured differently. If the organization allows frequent console changes, the process problem has to be fixed before the code can reliably represent reality.