How can I safely update a GitOps repository from concurrent deployment pipelines?
Asked by
MellowPine47
I recently started looking into Kustomize to organize application deployments managed by Argo CD. My planned flow is for CI to clone a GitOps repository, switch to the overlay for the target environment, run `kustomize edit set image` with the current commit SHA, then commit and push the change.
The concern is that several developers could trigger pipelines at nearly the same time. Each pipeline might clone the repository before the others have pushed, update the same files, and then encounter a rejected push or potentially create conflicting deployment updates. What are reliable ways to serialize these updates or otherwise handle concurrent deployment pipelines?
Answers
A common solution is to add concurrency control to the CI workflow. Use the deployment environment as the concurrency key, allow only one job for that environment to run at a time, and queue newer jobs instead of running them simultaneously. That prevents two pipelines from modifying and pushing the same overlay at once. You should still fetch or rebase immediately before pushing and handle a rejected push defensively, since serialization settings can be bypassed by manual changes or separate workflows.
Reply to this answer
Tools such as Kargo are designed for this broader GitOps promotion workflow. They can coordinate image updates and environment promotions rather than having every application pipeline directly edit and push the deployment repository. Kargo is not simply a wrapper around Kustomize; it is an orchestration and promotion layer that can use tools such as Kustomize as part of the process.
MellowPine47: Got it—so it handles promotion orchestration, while Kustomize is still the tool that renders or edits the manifests.
Reply to this answer
Add your answer
Answers are reviewed before publication.
MellowPine47: That makes sense. I’ll look into making the deployment job use a per-environment concurrency group so updates are queued instead of racing.