I'm managing around 30 microservices on Kubernetes and giving each developer their own namespaces, initially copied from production. However, when there's an update to a ConfigMap in production, some developers' environments don't reflect those changes. We've experimented with various solutions like sync scripts (which tend to break silently), Helm value overrides (which still drift), Kustomize overlays with ArgoCD (though it keeps drift version-controlled, it doesn't eliminate it), and a shared staging environment (which causes conflicts when two developers are testing at the same time). What strategies do other teams, especially those with over 20 services, employ to keep their dev environments in sync with production?
4 Answers
We have a similar structure with dedicated clusters for development, staging, and production. Each environment is deployed through the same pipeline but developers can only interact with the clusters to view the setup. This separation helps keep everything organized.
Are you asking whether developers are working in their local environments or if they have an entire K8s setup for development? If it’s the latter, enforcing deployments from the same branch will help keep configurations uniform, aside from environment-specific variables.,
I'm referring to each developer's environment running within K8s since managing all services locally is impractical. They each have a full service setup in their namespaces, using tools like Telepresence to manage traffic.
In our case, we maintain two separate environments: a staging environment accessible to all developers and a production environment that’s only updated weekly. Developers utilize the staging for their tests, deploying merged versions of their code there. Our team relies on a simple command that grabs the latest config from the staging environment, keeping things up-to-date without needing to synchronize too much data. It's also easier to manage since we work in a monorepo.
What happens when a service running in the staging environment relies on one you're developing locally?
It sounds like your setup might be overcomplicated for what developers actually need. Instead of requiring them to have full cluster setups, you might consider using test containers and mock data for local development. By doing this, each developer can run the specific service they are working on in isolation, using testcontainers for external dependencies like databases and message queues. It may not answer your question directly, but it helps mitigate drift since most of the dependencies are mocked.
Yeah, it seems like you might not need full end-to-end tests during local development. Using mocks could simplify things!
I got the impression this setup is intended for isolated dev environments, not directly on their machines.
That sounds a lot like our system, though our pipelines are distinct. Compliance measures mean we have to keep everything very structured.