I'm considering moving our services to distroless images. The smaller attack surface and reduced scanner noise are appealing, but these images usually lack a shell, package manager, and sometimes the libraries I'm used to. When a production issue occurs and logs or telemetry don't explain enough, what is your practical debugging workflow? Do you use ephemeral containers, a separately maintained debug image, namespace-sharing tools, host access, or something else?
4 Answers
The tooling should not be the only plan. Improve structured logs, traces, metrics, crash dumps, and runtime diagnostics so you can capture evidence without manually entering the container. For recurring problems, have the application collect a bounded diagnostic bundle or expose a tightly protected diagnostic operation. That lets you investigate safely and reproduce the issue elsewhere instead of relying on an interactive production shell.
The usual approach is to keep the production image minimal and launch a temporary toolbox beside it. In Kubernetes, an ephemeral debug container can share the target pod's process and network namespaces, giving you tools like a shell, strace, ss, and curl without adding them to the application image. A pinned development variant of the same image can make this repeatable, but it should be rebuilt and patched alongside the production image.
A separate debug image is a good break-glass option, but treat it like production software: build it from the same source and version as the application, scan it, and keep it patched. Embedding BusyBox or another shell directly into the production image works in a pinch, but it increases the shipped attack surface and undermines the reason for choosing distroless. A temporary debug container is generally cleaner.
With plain Docker, you can run a throwaway toolbox container that shares the target container's namespaces. For example, sharing the PID and network namespaces lets a diagnostic image inspect the running process and network without changing the shell-less application image. Give it only the capabilities required for the investigation, remove it afterward, and use host namespace tools such as nsenter only when you genuinely need access to the target filesystem or host-level details.
Sharing the process namespace is especially useful because you can inspect the live process rather than starting a separate copy. Just remember that the debugger has its own mount namespace, so files from the target may need to be viewed through /proc//root.