I'm part of a medium-sized company with around 150 employees, including a dedicated platform team and four security engineers. We run Kubernetes on EKS, using GitHub Actions to build images that get pushed to ECR. We utilize Grype for scanning on every pull request, blocking on critical and high vulnerabilities. However, despite our efforts, the number of Common Vulnerabilities and Exposures (CVEs) in our base images isn't decreasing. For instance, I recently pulled a fresh nginx:1.25 image, which had 140 CVEs right off the bat, most of which were for packages that shouldn't even be in a production runtime environment.
We've tried various methods, including multi-stage builds to strip unnecessary content and suppressing non-runtime reachable CVEs, but our security team prefers not solely relying on reachability for closure on findings. This means engineers are still busy triaging 80+ CVEs per sprint whenever there's a new upstream image.
I'm curious about what strategies other organizations are using to maintain their base images. Do you create your own base images from scratch, use hardened images with some SLA, or find a balance? Specifically, I'm looking for actionable changes that impact the actual CVE counts, not just improved visibility.
7 Answers
Years ago, we relied on official RHEL images for our software, which worked well since it kept OS-related CVEs out of our control. If I were to do it again, I'd go back to a well-maintained OS image to control the application vulnerabilities better.
We've started moving to distroless images and recommend building them from the source or importing them into our own repository. Google provides some good options to start with, but it really is the best way to minimize CVEs in production.
Exactly, distroless and hardened images should be used together because not everything fits neatly into just one category.
Check out docker hardened images; they’re available for free. But who knows how long that will last with their need to monetize.
We dedicate part of our engineering team, about 15 of 70 people, every six weeks to fix CVEs from Aikido. Our SecOps determines the priorities, and this approach has worked well so far. We address the most critical issues quickly, but it’s an ongoing effort.
We use Chainguard for our images. While we tried filtering out unreachable findings, we opted for paid solutions because it saves time on audits and frees up our team from constantly patching images. Sure, you can do it yourself, but it becomes a full-time job if your list of images is extensive. I believe it's more about adjusting the mindset regarding security in a container environment compared to traditional servers.
Glad to see Chainguard mentioned. They really do the hard work of patching, unlike competitors who shy away from it.
Are there benefits to using the free images? What’s missing that makes paid ones better?
We've moved from using Bitnami to a system where it varies. Some official images are solid, while others come with security issues. For those that aren't patched, we create our own Alpine containers and install software ourselves. It's straightforward for some apps like nginx, but others require some custom scripting. I've heard of companies paying for hardened images from places like Chainguard, but I haven't tried those myself.
I got a quote from Bitnami for 60k to access an image, and I'm like, no way! I could hire someone for way less to make our own.
Honestly, any vendor selling hardened images that aren't distroless isn't worth it. Bitnami's images are either outdated or poorly secured.
But distroless images can't solve all our problems, especially if dependencies rely on libraries like glibc.