Have You Struggled with CI/CD Pipeline Bottlenecks?

To tackle the issue differently, I’m looking into using ephemeral build containers. By having Jenkins spawn builders as pods in a Kubernetes cluster, we can delete the pod and its data after the image is built and pushed. This helps manage disk space and merges our build infrastructure, but just keep in mind that you’d need to handle docker artifacts on your host.

If you're not already using it, I recommend exploring GitHub Actions. We’ve implemented Otel Traces that can really help with debugging these issues. It boosts observability and lets you split up tasks. More CI systems should embrace this kind of tracing, I believe.

I can relate to your situation – oversized or misconfigured images can really slow things down. What worked for us long term was making our images leaner from the start. We use RapidFort, which automatically trims unused stuff, and they offer hardened near-zero CVE images. This strategy not only saves us from having to deal with junk but also speeds up our Jenkins builds significantly.

It sounds like you ran into a common issue with handling images. When you build images on your own runners, Docker can end up keeping old images until you run out of disk space. It’s a tough lesson, but it’s one that many DevOps folks learn the hard way. To manage this better, consider setting up cron jobs that automatically prune old images and stopped containers regularly. Just be careful not to overdo it, since that could slow down your builds when they have to pull everything from scratch again.

You’re definitely not alone in this struggle! We faced similar problems in multiple projects because of old images hogging disk space. We set up cron jobs for regular pruning, and it made a big difference. But make sure to monitor the CI’s performance after pruning – daily or weekly pruning can sometimes lead to slowness.