Hey all, I'm struggling with our security measures for Kubernetes. Our existing SAST/SCA tools handle image scans during CI pretty well, but once it hits runtime, we lose visibility on vulnerabilities.
We're dealing with a lot of init containers, sidecar proxies, and temporary jobs that quickly spin up and down. Some of these pull from images we've never scanned, and others run with elevated privileges that we overlooked during our static analysis.
Just last week, we discovered a vulnerability in a logging sidecar that our pre-deployment scans completely missed, all because it was injected by our service mesh.
How do others in the community gain insight into the actual attack surface of running pods rather than just relying on what was scanned in CI? I appreciate any tips you can share!
3 Answers
What if you implemented a policy engine or a validating admission webhook that only allows images from your secure internal registry? On that registry, you could ensure that all images are scanned both during uploads and at regular intervals.
CI-only scanning tends to fall short when service meshes and injected components come into play. Init containers and sidecars aren't part of the build process, so static tools will miss them. The main issue is the disconnect between what was approved in CI and what's running in the cluster. Keeping an inventory of injected containers and ephemeral jobs at runtime can help spot those blind spots, but the real challenge is linking this runtime reality back to the original intentions and ownership.
Consider using an open-source security scanner like Trivy in your cluster. It can scan everything that pops up, even those components injected by service meshes.
That would definitely help with controlling provenance, but it doesn't fully solve the problem of visibility for what's running once injection and ephemeral containers are involved.