We're dealing with several teams that need to set up their own EKS/AKS clusters, so we've created some Terraform blueprints with built-in best practices. These serve as a solid base for them to easily deploy clusters. However, the issue arises when teams clone the blueprints and start making their own customizations; they seldom update their copies with our latest fixes, improvements, or new policies. Over time, this leads to disparate versions that drift away from our intended standards. I'm curious about how others handle this situation. Do you enforce any type of sync or upgrade policy? Do you manage versions through modules, or have you accepted the chaos?
6 Answers
I've faced similar challenges. You need to clarify your role: are you just a resource provider, or are you responsible for all clusters? If it’s the former, focus on communication about the benefits of keeping clusters updated. If it’s the latter, set clear rules, potentially updating each cluster yourself or imposing deadlines to meet standards. I usually take the latter approach to handle upgrades.
I believe the problem is that you're allowing teams to modify the templates instead of just customizing approved parameters. Only expose a 'black box' to other teams. They should have access to read the code but not change it directly. This keeps your team accountable and avoids messy configurations.
Your team needs to decide on your role regarding these updates. If you're merely providing blueprints, your communication should shift to being a support model. If you're responsible for security, lock your templates in a repository that requires approval for any changes. Cloning templates into different repos isn’t scalable for enforcing updates. Instead, create changelogs for your teams, and consider using tools that help automate this process and keep them informed.
Consider implementing GitOps. It adds some complexity, but it ensures you have a single source of truth. You can consolidate your Terraform code in a repo where teams can submit cluster requests. They pull a PR when ready, and you approve it before Terraform runs automatically. For updates, the same process applies, ensuring everyone works off the same foundation.
Yeah, tools like Argo or Flux are great for that kind of workflow!
Do you mean using a Terraform operator with GitOps? How would you suggest pairing those two?
This issue is typical in platform engineering. Are teams really just copy-pasting your Terraform modules and modifying them? It's not sustainable. One team should maintain those templates, and if changes are needed, they should follow a structured request process, as mentioned in the top comment.
We tackled this by creating a Terraform module that no one can modify. They can request changes, but modifications are handled by our team only. We also use GitOps, where Terraform spins up the clusters and resources, while core applications are managed through tools like Argo. This way, teams can still deploy in their own namespaces without chaos.
Exactly! Following design patterns is crucial. No one should be modifying core modules.