I'm looking for advice on building a development environment in my private Kubernetes cluster, where I have a variety of containers deployed. I need a universal way to impersonate these containers using a development pod—ideally, I want to integrate source files, a debugger, and a connected IDE like JetBrains or VS Code. The challenge is compounded by the complex configuration of the pods, which include numerous environment variables and vault secrets. I develop on a Mac with an M processor, and I've encountered issues with certain applications that can't compile on ARM, which prevents me from using tools like mirrord effectively. What I'd really like is to customize any source image with devcontainer.json to include development tools and packages, and then deploy it into my cluster. So far, I've used DevPod and DevSpace for file synchronization, but I'm running into limitations since DevPod is no longer maintained and setting everything up manually is a hassle. Are there any better alternatives?
5 Answers
I recommend going full Kubernetes-native instead of fighting with DevPod. You can create a small Helm chart to spin up a dev pod from any existing deployment, which copies the container's environment variables and mounts secrets in a read-only mode. You can then install your required dev image with SSH and language tooling, and use JetBrains Gateway or VS Code Remote-SSH to connect. For syncing files, tools like Mutagen or DevSpace sync work great, so you'll have a live editable environment inside your cluster. For handling secrets, consider using Infisical; it keeps your dev pod in sync with production secrets and automatically updates them when they rotate.
We manage this via ArgoCD Application Set. When a developer labels their PR, ArgoCD creates an application for that specific PR. We even have a process to build the image with the commit ID, so everything aligns with the PR. This way, we tackle environment variable issues because the same config is shared as in dev. Additionally, you get complete access to a container in your namespace just for the pull request. Once the PR is merged or closed, ArgoCD cleans everything up automatically, which keeps things tidy.
Do you face any issues with build times when a dev is trying to iterate quickly? Seems like a cool solution, but does it require an additional process for faster iterations?
One thing you can also do is use minimal container images to speed up build and deployment times with tools like Minimus.
We created Okteto for exactly this purpose. By using our CLI, you can run `okteto up` and begin developing directly in your Kubernetes environment with all configurations and dependencies in place, which eliminates the hassle of local setups. Check it out and let me know if you need help getting started!
You could also take a look at using 'Coder' with a Kubernetes deployment type. It gives you a full VS Code environment with terminal access and even nested Docker support, which could really streamline your workflow!
I saw you mentioned issues with mirrord since you're on an ARM Mac. Have you considered using a devcontainer in combination with mirrord? The devcontainer can be emulated if ARM isn't supported, which might help.
But what’s the need for this setup? What problems does it solve?