I've transitioned from managing a single multi-tenant deployment to over 200 single-tenant customer environments over the past few years. Initially, GitOps was effective, but as the scale increased, we started facing several challenges such as delays due to pull request queues, emergency fixes causing environment drift, and issues with a single problematic environment blocking large rollouts. We found ourselves needing additional orchestration tools outside of Git to address these issues. I'm interested in learning how others are coordinating rollouts and handling drift reconciliation at such a large scale.
5 Answers
We tackle rollout issues by shipping metrics to S3 and then from S3 to VictoriaMetrics. That way, we can monitor ArgoCD and app statuses through dashboards and quickly identify any drift or version mismatches.
We've developed a tool that focuses on managing deployments through Helm charts. It facilitates approvals and guarantees that each environment reflects what you've authorized for deployment. If you're interested, I can set up a demo!
The shift from multi-tenant to single-tenant deployments often comes down to customer demand for isolation due to security concerns. It's tricky, as the challenges don't disappear, but at least you address specific needs.
Have you considered using ArgoCD's progressive sync feature? It’s a great way to manage rollout flows more efficiently.
It sounds like you’ve reached a point where GitOps might be more of a hindrance than a help. Check out Cycleio; it treats environments as first-class objects, effectively addressing the PR overhead and simplifying orchestration.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures