How Can You Track Deployments Across Different Environments?
Asked by
CloudyCoder92
I'm working with multiple repositories for services across environments like DEV, QA, UAT, and PROD, using ArgoCD for deployments. While GitOps should theoretically be our source of truth, I've noticed several issues: image tags sometimes don't match the actual runtime, some CI builds are never promoted, release branches don't consistently align with the environment state, and there's no clear history of releases per environment. It seems like while GitOps tells us what should be deployed, it doesn't necessarily give a clear picture of what actually occurred over time. I'm looking for insights on how others manage release tracking and auditability across environments. Do you build your own solutions for release tracking, or do you trust the history provided by Argo or Flux?
Answers
That's precisely why I’m transitioning from a pull model to a push model. While a pure pull model simplifies automation, it can strip away too much control over what's actually happening.
Reply to this answer
You can set up FluxCD to automatically correct any drift you notice. I believe Argo has a similar feature. Beyond that, proper permission settings are key to preventing drift, ensuring everything goes through Git. If you're allowing manual interventions, consider implementing quotas and regular cleanups to keep things in check.
Reply to this answer
If your cluster's state is drifting from your repo setup, it suggests a configuration issue. You can tighten permissions on your kubectl users, ensuring only Argo can create resources. Additionally, setting up webhooks in GitHub or GitLab can notify Argo of changes that should trigger a sync.
Reply to this answer
Whenever expectations and reality don’t quite align in your release process, treat it as an incident to investigate and fix the root cause. All the issues you mentioned indicate potential configurations or design problems that need addressing.
Reply to this answer
Switching to SHA references for your images can really help avoid drift. By using a SHA instead of a simple tag, it’s like pinning a specific commit. We use kaniko for our builds, which generates an env file with those SHA references to make deployments smooth.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions