I'm relatively inexperienced with Kubernetes and am trying to understand a recurring problem in our bare-metal cluster. After a previous containerd-related change, pods occasionally became stuck in ContainerCreating. A costly workaround was added: a scheduled job that restarts Calico and runs a test pod every night, but several of those jobs have themselves been stuck for days.
During a recent backend deployment, some of my application pods had ImagePullBackOff because I used the wrong image tags, but even Valkey became stuck in ContainerCreating. The affected nodes appear to have broken or inconsistent CNI state. Calico is running docker.io/calico/node:v3.23.5 while the Kubernetes server is v1.35.0, and I'm unsure whether that combination is supported.
Calico has also reported that it cannot automatically detect the host MTU because no interface matches its pattern. The main interface on the machines is named enX0, while the configuration may be expecting something like eth0. I also suspected an IPAM problem, although the address pool appears to have plenty of free space.
What would be a systematic way to diagnose this? Which logs and configuration should I inspect, and what would be a proper long-term fix instead of periodically restarting Calico? Would upgrading Calico, rebuilding the cluster, or moving to a managed Kubernetes installation be more sensible?
2 Answers
Consider using a Kubernetes-focused terminal dashboard such as k9s to make node, pod, event, and log inspection easier, but treat it as an interface rather than a diagnosis. The important part is still comparing a working worker to worker-2 and recording the failure before deleting anything. AI tools can help interpret output, but avoid allowing them to execute destructive commands automatically, especially in a cluster containing stateful services.
Start with kubectl describe pod on an affected pod, especially the Events section. Messages such as FailedCreatePodSandbox usually point directly at the CNI or runtime. Then compare a healthy and affected node with kubectl get nodes -o wide and inspect the Calico pod logs for the relevant node. Kubelet logs around the creation attempt are also important: journalctl -u kubelet --since "30 min ago".
Check the Calico CNI files under /etc/cni/net.d/, verify that the referenced binaries exist, and inspect the IP pools and allocations. An IPAM command such as calicoctl ipam show can confirm whether addresses are actually exhausted. Also review the Calico configuration for the MTU interface pattern; it needs to match the real host interface rather than assuming eth0.
Do not treat a recurring daemon-set restart as a fix. First establish whether this is an unsupported Calico/Kubernetes combination, a stale node-level CNI state, an MTU mismatch, or a kubelet/container-runtime issue. Upgrade only after checking the supported version matrix and testing the change.
The Calico daemonset and controller are currently Running, but several old token-refresh jobs remain stuck. The affected Valkey pod is scheduled on worker-2, has Calico annotations but no assigned pod IP, and its describe output has no Events. The IP pool shows roughly 128 addresses in use out of about 65,000, so exhaustion seems unlikely. I’ll compare the node’s CNI files, Calico logs, and kubelet journal with a healthy worker.