What’s the Right Balance for Minimal Production Containers?

When we switched to distroless images, our ops team struggled because they couldn’t do anything. So, we ended up creating debug images with shells and other tools to help during development and quickly resolve issues. Having some debugging tools can be useful!

I’ve found that rushing to make the smallest image isn’t always worth it. Start with the dependencies you know your app needs and run your CI/CD scripts locally against your candidate image. Take note whenever something fails due to a missing tool and discuss whether it’s necessary for production or just for builds. Avoid including build tools in production images, but having basic utilities handy can really help during live debug sessions.

You can strip away anything unnecessary when creating minimal images. With the debug container support in Kubernetes, you typically don’t need to include a lot of tools directly in your images. Instead of using these images for CI/CD, consider creating specific build images with the necessary tools included.

Keep it simple: essential items are what your app needs to run; optional tools are the rest. For production, don’t install any unnecessary packages. Consider using multi-stage builds where you can keep development tools separate from production images. I'm curious about the specific sizing issues that led to your question.

CI/CD shouldn't really be slowed down by the size of your final images. When deciding what's essential, focus on what's absolutely necessary to run your service smoothly. Everything else can be considered optional. But it's a balance—think about storage and network costs, startup times for new pods, and how much work it takes to keep images slim. It really depends on the stack you're using as well; for instance, Go apps tend to run really small.

It's definitely challenging to create super minimal images. An alternative is to use vulnerability-free container images or explore distroless options. They can be cleaner or easier to manage with a package manager, so it's worth checking them out.