We run EKS with ArgoCD, Helm, and GitLab. Our GitOps repository has separate directories and values files for dev, SIT, and UAT. Secrets are already managed through AWS Secrets Manager; this question is only about non-secret environment variables.
Developers frequently adjust variables in the dev configuration while testing. When the change is ready for SIT and later UAT, someone manually copies the relevant values into the next environment's file. This regularly leads to missed variables, formatting mistakes, configuration drift, and broken deployments.
The proposed solutions are either separate branches for each environment, promoting changes from dev to SIT to UAT, or moving the values to AWS Parameter Store. I'm concerned that environment branches would create merge conflicts and accidentally promote environment-specific endpoints, hosts, or IRSA role ARNs. Parameter Store seems to move the same manual copying process outside Git, where it would have less reviewability and visibility.
What repository structure and promotion workflow have worked well for you? How do you promote the same application change while preserving genuinely environment-specific values, and what would you choose if designing the system again?
5 Answers
Parameter Store can be useful for runtime configuration, but it does not automatically solve promotion. If values need review, auditing, and reproducibility, keep the desired state in Git or in versioned artifacts and use Parameter Store only for values that genuinely belong outside the repository. Also separate frequently changed application settings from infrastructure identity and connection details; putting everything into one large environment-variable file makes omissions and accidental changes much more likely.
Versioned OCI artifacts are another clean option. CI can package the rendered deployment tree or Helm-related configuration as an immutable artifact. Each environment stores only the artifact version it should run, while its local overlay supplies values that must remain different. Promotion becomes updating a version reference rather than copying YAML between directories, and rollback is straightforward because older versions remain available.
Treat the application configuration as part of the deployable version rather than manually copying values between environment files. The application repository or CI pipeline can generate and publish the rendered manifests, or at least a versioned configuration artifact, and the GitOps repository promotes that same artifact through dev, SIT, and UAT. Each environment can still provide its own infrastructure-specific overrides, such as URLs, role ARNs, and certificates. This gives you a reviewable diff and ensures the tested artifact is the one promoted everywhere.
A promotion controller such as Kargo can automate the sequence. It can detect a new image or configuration artifact, open a change for the lowest environment, wait for deployment and health checks, and then promote the same version to the next environment. This provides progressive delivery and cluster validation, though it introduces another controller and may create several approval changes to manage. It is worth evaluating if you want automated promotion gates rather than only a repository convention.
This approach works well, but the number of generated changes can become cumbersome with many environments. Make sure the approval and validation workflow is simpler than the manual process it replaces.
Avoid environment branches. They tend to create merge-conflict and cherry-pick problems, especially when values are intentionally different. A better pattern is a shared defaults file plus small environment overlays. Separate application configuration from infrastructure configuration: promote application settings through the environments, while keeping endpoints, DNS names, resource sizing, and IRSA details in environment-owned overlays. Add CI validation to ensure required keys exist, values have the expected type and format, and every overlay is rendered successfully with Helm.
Layering only helps if the boundaries are clear. Defaults should contain safe values, application overlays should contain settings that are meant to progress, and infrastructure-specific values should remain owned by each environment. Validation can also check that every required default is either inherited or explicitly overridden.
Our pipeline publishes rendered manifests to the GitOps repository. Promotion then changes the artifact reference in the next environment instead of copying individual variables. ArgoCD applies the result, and the rendered output can be diffed before promotion.