How Can You Track Deployments Across Different Environments?

Switching to SHA references for your images can really help avoid drift. By using a SHA instead of a simple tag, it’s like pinning a specific commit. We use kaniko for our builds, which generates an env file with those SHA references to make deployments smooth.

Whenever expectations and reality don’t quite align in your release process, treat it as an incident to investigate and fix the root cause. All the issues you mentioned indicate potential configurations or design problems that need addressing.

If your cluster's state is drifting from your repo setup, it suggests a configuration issue. You can tighten permissions on your kubectl users, ensuring only Argo can create resources. Additionally, setting up webhooks in GitHub or GitLab can notify Argo of changes that should trigger a sync.

You can set up FluxCD to automatically correct any drift you notice. I believe Argo has a similar feature. Beyond that, proper permission settings are key to preventing drift, ensuring everything goes through Git. If you're allowing manual interventions, consider implementing quotas and regular cleanups to keep things in check.

That's precisely why I’m transitioning from a pull model to a push model. While a pure pull model simplifies automation, it can strip away too much control over what's actually happening.