There's a common belief that in Kubernetes, containers should be stateless. You're advised to use a Persistent Volume if you need to save data. This approach works well for production microservices, but when it comes to a Development Environment, I'm finding it more challenging.
Here's my dilemma: if I treat a Kubernetes pod like a "Cloud Workstation," I can mount a Persistent Volume to save code which is great. However, I run into issues with system dependencies. If I need to install a library or tweak system files for debugging, those changes disappear the moment the pod restarts, as they're made in the container's ephemeral filesystem.
Just updating the Dockerfile isn't practical in a development loop. It disrupts the flow to have to rebuild the image every time I want to make a small change. So my question is: is Kubernetes not well-suited for this kind of "Stateful Pet" approach, or are there modern solutions or technologies I might not be aware of? I want to know how others are handling things like root filesystem persistence or recreating the full state of their development environments, especially in the context of platforms like Coder, Gitpod, or Codespaces.
3 Answers
I don't really think this is a big issue. When you require a dependency for more than just a quick test, you should add it to the Dockerfile instead of letting it remain undocumented in the dev setup. Plus, if you're unsure about a dependency, you can still install it directly and test before a restart. For me, I'd be more worried about work interruptions from Kubernetes evictions than this persistence hassle.
Have you checked out the devcontainer system from Codespaces? It's another container build designed specifically for development. There are plenty of pre-configured options for different programming languages. You can also run one-off commands without having to take on the burden of managing the entire image. It streamlines the process a bit!
I've been using web-based IDEs on Kubernetes, like Codeserver. Here's what works for me: I keep my 'workspace data' backed by Persistent Volumes, and I bake in important system libraries directly into the container images. For project-specific libraries, I contain them within project containers. When I'm building and running my app, I always do it within a container that mimics production as closely as possible. AGreat tip is to avoid just installing packages directly in the IDE. Instead, use a controlled container process. You can also set your PATH to include directories on a PVC, but be cautious about compatibility issues. My advice is to pre-configure your web IDE container to handle package installations properly.
True, but it feels like Kubernetes isn't really used to schedule dev containers like it should be.