Imagine Terraform is running in production with an S3 backend for state storage and a DynamoDB table for state locking. A developer accidentally deletes the lock table, and two engineers run terraform apply at roughly the same time. Later, the infrastructure appears inconsistent. What would actually happen internally? Could the state or infrastructure become corrupted? How should the team recover safely, and when would restoring an S3 state version, importing resources, or manually editing state be appropriate? What controls would prevent this situation from happening again?
4 Answers
If locking was bypassed or a forced unlock left Terraform running without coordination, both processes could read the same S3 state and then write updated state independently. The later state upload could overwrite the earlier one, causing a last-writer-wins situation. That is usually state divergence or lost state updates rather than Terraform magically corrupting every resource. The real infrastructure might still contain changes from both runs, while the state records only one view of them.
The prevention plan matters more than the recovery trick: use CI/CD with serialized production applies instead of letting engineers apply from laptops, restrict production write access, deny deletion of the backend bucket, state objects, and lock table, and require approval for backend changes. Enable S3 versioning, encryption, access logging, and recovery controls. Terraform now supports an S3-native lock file, so consider using that instead of DynamoDB where appropriate, but do not assume changing the locking mechanism alone solves access-control or concurrent-pipeline problems.
Also treat the backend resources as a separate, tightly protected stack. The people who can deploy application infrastructure should not automatically be able to delete the state bucket or locking mechanism.
The premise needs a little clarification: if Terraform cannot access the DynamoDB table while trying to acquire a lock, it will normally fail before making changes. A missing table should produce a backend or lock-acquisition error, not silently allow both applies to continue. You can test this by denying access to the table and running an apply in a non-production environment.
Recover conservatively. Stop all applies, preserve the current S3 object and its versions, inspect CloudTrail and Terraform logs, and compare the state history with the actual infrastructure. If S3 versioning is enabled, restore the last known-good state only after confirming which resources it represents, then run terraform plan. Do not blindly restore state, import everything, or edit the state by hand. Import is for resources that exist but are missing from state; manual state editing should be a last resort. Recreate the lock table with the correct schema if it is still required, verify permissions and stale locks, then reinitialize and apply only after the team agrees on the desired state.
That is the important failure mode to investigate, but first confirm whether locking was actually bypassed. With the table simply missing, standard Terraform behavior should be to stop rather than proceed unlocked.