I've been diving deep into distroless containers for our security measures, and while they sound excellent on paper—smaller attack surfaces and fewer CVEs—implementing them has been challenging. The lack of a package manager means I have to rewrite all Dockerfiles or juggle dual images. This has led to pipeline breaks since I can't install debugging tools or perform basic tasks in production without a shell.
Our security team wants minimal images, but the development team is struggling to deliver features without getting bogged down in Docker-related issues. We've tried multi-stage builds using Ubuntu or Alpine for building, but that makes our CI/CD pipelines painfully slow and requires constant updates when base images change.
Plus, there's the challenge of debugging in production. With distroless containers, I can't exec in to troubleshoot. I essentially need a separate debug image for any troubleshooting. So how are others managing this? What's your workflow for keeping useful build tools while still deploying lean and secure runtime images? Is there any tool that simplifies this process or are we all just enduring the struggle? We're on AWS ECS, and security keeps flagging CVEs in our Ubuntu images, so switching to distroless feels like trading one set of problems for another.
8 Answers
If you're looking for a solution within ECS, have you tried `docker debug`? It can be a great way to access the machine running your service, assuming you have that ability.
Have you looked into attaching ephemeral debugging containers? They're a neat way to troubleshoot without needing a full separate setup. Check out the Kubernetes docs on ephemeral containers for more details!
Honestly, this feels like a classic AI-generated post. Not sure how people are responding to it.
To effectively use distroless, it’s essential to maintain strong auditing and telemetry while keeping a second image for debugging. In today’s fast-paced environment, having robust applications that can handle failures without constant maintenance is key. You should plan for scheduled maintenance to manage CVEs, but automating the fixing would alleviate some stress as vulnerabilities come up.
Your builds shouldn’t take that much longer when using multi-stage builds. If they are, look into the performance of your CI runners. Also, for debugging in a production setup, using `ubuntu:latest` as an ephemeral debug container in Kubernetes can be a lifesaver!
The `kubectl debug` command is super helpful! It lets you spin up temporary containers as part of a running pod, which is perfect for debugging on the fly.
For me, I just go with `FROM scratch`. The key is to have your application expose profiling and debugging APIs that you can access over the network. Alternatively, you might consider using a shared volume with a busybox binary to get a shell just in time for debugging when you need to use ECS exec.
Using an APM tool and sending OpenTelemetry from your containers can really help with debugging. It’s generally considered an anti-pattern to try and debug inside a running app like we did years ago. Better logging and monitoring should reduce the need to access containers in the first place.
Sounds good in theory, but sometimes you need to get in there and see what's going on.
Couldn't agree more! Solid logging, metrics, and tracing can eliminate a lot of the need for direct container access.