I'm building a proof of concept that needs to test a DaemonSet or node agent against a Containerd CRI setup close to what runs on AWS EKS managed nodes. Minikube and Kind are not suitable because they abstract away too much of the node and do not accurately reflect custom Containerd configuration or host-level behavior. Running a managed EKS cluster for extended debugging would also be too expensive. What local or inexpensive setup would provide the most realistic EKS-like node environment for deep runtime and CRI testing? If you have used a similar approach, what differences did you find when moving from the local environment to actual EKS?
4 Answers
You can run an Amazon Linux or EKS Anywhere virtual-machine image under QEMU and customize it directly. That lets you inspect the host, install the same kind of runtime components, and experiment with node-level services without paying for a continuously running cloud cluster. It still will not reproduce every EC2 or managed-control-plane detail, so treat it as a close node-level approximation rather than a perfect clone.
A custom node image can make Kind or another local tool closer to the target, but EKS-specific behavior is difficult to reproduce completely. Cloud integrations, instance metadata, networking, permissions, bootstrap logic, and managed-node updates can all introduce differences. I’d use a VM-based Containerd cluster for most iteration, then maintain a very small real EKS test cluster for final compatibility checks.
EKS Anywhere may be worth evaluating if you want an AWS-oriented Kubernetes distribution that runs outside the managed EKS service. It can provide a more relevant baseline than a generic local cluster, although you should still compare the actual Containerd configuration, bootstrap scripts, networking, IAM integration, and kernel behavior with production EKS nodes.
A practical approach is to create a small scratch cluster on one or more virtual machines using kubeadm and Containerd directly. Start with an Amazon Linux 2023 image, then use the publicly available EKS node-image build configuration as a reference for packages, system services, kernel settings, Containerd configuration, and bootstrap behavior. This gives you much more control than Minikube or Kind, while keeping the test environment inexpensive.
That sounds like the best balance for my use case. I mainly need to validate the node agent and CRI behavior before testing the final version in EKS.