We're setting up a Kubernetes fleet on CloudStack and need to build our own node images. For now, we're using Kubernetes image-builder with Ubuntu as the base, but vulnerability scanners such as Trivy report many findings that appear to be noise, including obsolete Linux headers that remain installed. We use Cluster API and want to establish a sensible hardening process. What tools and base operating systems are you using for Kubernetes node images, and what steps do you take to reduce vulnerabilities and unnecessary packages?
5 Answers
If your environment supports it, Talos is worth considering because it has a very small, purpose-built operating system and greatly reduces the amount of OS maintenance and hardening you need to manage.
A common approach is Packer combined with the CIS Ansible hardening playbooks. Ubuntu 22.04 LTS is a reasonable base, and you can remove unused Snap packages and cloud-init components after the initial provisioning. If Trivy is flagging old kernel headers or images, explicitly purge packages that aren’t needed in the final node image; that usually removes a lot of the noise.
Bottlerocket is also a strong choice for Kubernetes nodes when it is compatible with your Cluster API and CloudStack setup. Its minimal, immutable design reduces the attack surface compared with a general-purpose distribution, though it may require adapting your operational tooling.
We use Debian for our golden Kubernetes images, built with Packer and an Ansible playbook. The playbook updates the kernel, removes anything the nodes don’t need, applies the required kernel parameters, installs containerd and kubelet, and generates an SBOM as the final step. The resulting image is under 700 MB, or roughly 300 MB compressed, and boots quickly before joining the cluster through cloud-init.
Diskimage-builder is another option if you only need to produce disk artifacts such as QCOW2 images. It can be faster and more direct than a full image-builder workflow, since the output doesn’t require a separate cloud platform to create it.
A build taking 20–30 minutes doesn’t seem especially problematic to me, but the simpler artifact-only workflow could still be useful if the extra image-builder steps aren’t needed.
That sounds appealing, but we’re using Cluster API, and the Talos providers we were considering are no longer the preferred option, so we may need to stay with a more conventional image pipeline.