I'm on a three-person team building a prototype AI tool: one data scientist and two engineers. Between the two engineers, we have roughly a year of combined Python experience. I have about two years of IT experience and recently had to learn Docker, GitLab CI, and software engineering practices for this project.
Our previous workflow used Docker Compose. We had a development deployment with a hot-mounted source directory, environment files based on a checked-in template, and Makefile commands. Commits triggered image builds, while staging and production used prebuilt images stored in internal registries and deployed through separate Compose configurations.
We recently lost access to our on-premises server and now have to deploy to a k3s cluster managed by another team. That team is also relatively new to Kubernetes. I learned enough about Helm and Argo CD to deploy the service, and created an infrastructure repository containing Kubernetes manifests, charts, and environment-specific values files.
Staging and production seem manageable because our CI runners build images, update image tags, and change the relevant namespaces in the infrastructure repository. The part I'm unsure about is development. We don't yet have direct cluster or namespace access, and I don't know what the intended workflow should be for each engineer to run an isolated development pod, view logs, and make quick changes without rebuilding and redeploying everything manually.
What would be a practical development workflow for a small, inexperienced team operating on a k3s cluster? Should we use local Kubernetes clusters, per-developer namespaces, tools such as Skaffold or Tilt, or something else?
3 Answers
Before adding more Kubernetes tooling, question whether Kubernetes is necessary for the prototype. It may be more operational complexity than a three-person team needs. If the organization has already committed to k3s, though, treat it as an infrastructure constraint and ask the platform team for a clearly defined developer contract: who owns namespaces, how deployments are requested, where logs go, how secrets are handled, and what level of access developers receive.
For learning and early development, run a local kind cluster. It’s easy to create and destroy, and you can test your Helm charts, services, networking, and configuration without waiting on the shared cluster. You can build the image locally and load it into kind, then use a local values file that sets the image pull policy appropriately. This gives each engineer an isolated environment and keeps experimentation away from staging.
I’ve started looking into kind and it seems like a good way to connect the Kubernetes concepts with our actual charts. The main gap is figuring out what should eventually run locally versus what needs to be tested in the shared cluster.
Once the platform team is ready, the usual shared-cluster pattern is one namespace per developer or per feature branch. Each namespace gets its own Helm release and resource limits, with CI building a development image and updating that release. A tool such as Skaffold, Tilt, or DevSpace can watch source changes, rebuild the image, and redeploy automatically, which is much closer to the Docker Compose hot-reload experience.
Keep staging and production separate from this workflow. Developers should not be editing production values directly; instead, use a small development values file or an automatically generated preview environment. Also make sure the cluster team provides centralized logs and a way to inspect pod events, since developers shouldn’t need unrestricted cluster-admin access just to debug an application.
Per-developer namespaces sound like the cleanest long-term approach, but they’ll require the platform team to provide quotas, permissions, ingress or port-forwarding rules, and a cleanup policy for abandoned environments.
That makes sense. We’re already committed to the cluster for now, so I’m mainly trying to establish a workable process while the infrastructure team finishes setting up access and ownership.