I'm relatively inexperienced with Kubernetes and am trying to understand a recurring problem in our bare-metal cluster. After a previous containerd-related change, pods occasionally became stuck in ContainerCreating. A costly workaround was added: a scheduled job that restarts Calico and runs a test pod every night, but several of those jobs have themselves been stuck for days.
During a recent backend deployment, some of my application pods had ImagePullBackOff because I used the wrong image tags, but even Valkey became stuck in ContainerCreating. The affected nodes appear to have broken or inconsistent CNI state. Calico is running docker.io/calico/node:v3.23.5 while the Kubernetes server is v1.35.0, and I'm unsure whether that combination is supported.
Calico has also reported that it cannot automatically detect the host MTU because no interface matches its pattern. The main interface on the machines is named enX0, while the configuration may be expecting something like eth0. I also suspected an IPAM problem, although the address pool appears to have plenty of free space.
What would be a systematic way to diagnose this? Which logs and configuration should I inspect, and what would be a proper long-term fix instead of periodically restarting Calico? Would upgrading Calico, rebuilding the cluster, or moving to a managed Kubernetes installation be more sensible?
4 Answers
The operational issue may be bigger than one broken pod. A cluster that depends on a daily 2 a.m. Calico restart is not healthy, and leaving old jobs stuck for weeks makes troubleshooting much harder. Document the exact Kubernetes, containerd, Calico, CNI, and host operating-system versions, then reproduce the problem on one node while collecting events and logs.
If only a few workers are affected and the workloads are portable, cordon and drain those nodes, remove their stale CNI/runtime state using a documented procedure, and rejoin or replace them. Do this carefully and verify storage, disruption budgets, and persistent workloads first. If the cluster was assembled without a clear maintenance plan, rebuilding it from a known-good design may be cheaper and safer than continuing to patch it.
The MTU warning is worth investigating, but it does not by itself prove that MTU is the cause. Confirm the host interface and its MTU with ip link, check the effective Calico MTU settings, and make sure the overlay overhead is accounted for. A wrong MTU generally causes connectivity or packet-fragmentation problems, while a pod stuck before startup can also be caused by failure to create the network sandbox, stale CNI files, or a runtime problem.
The pod output showing Calico annotations but no IP is useful evidence that networking was partially initialized. Look at the node-local Calico CNI log directory and the container runtime logs at the exact pod creation time. Also inspect the events for the namespace and node, since deleting the pod may have removed the most useful evidence.
Consider using a Kubernetes-focused terminal dashboard such as k9s to make node, pod, event, and log inspection easier, but treat it as an interface rather than a diagnosis. The important part is still comparing a working worker to worker-2 and recording the failure before deleting anything. AI tools can help interpret output, but avoid allowing them to execute destructive commands automatically, especially in a cluster containing stateful services.
Start with kubectl describe pod on an affected pod, especially the Events section. Messages such as FailedCreatePodSandbox usually point directly at the CNI or runtime. Then compare a healthy and affected node with kubectl get nodes -o wide and inspect the Calico pod logs for the relevant node. Kubelet logs around the creation attempt are also important: journalctl -u kubelet --since "30 min ago".
Check the Calico CNI files under /etc/cni/net.d/, verify that the referenced binaries exist, and inspect the IP pools and allocations. An IPAM command such as calicoctl ipam show can confirm whether addresses are actually exhausted. Also review the Calico configuration for the MTU interface pattern; it needs to match the real host interface rather than assuming eth0.
Do not treat a recurring daemon-set restart as a fix. First establish whether this is an unsupported Calico/Kubernetes combination, a stale node-level CNI state, an MTU mismatch, or a kubelet/container-runtime issue. Upgrade only after checking the supported version matrix and testing the change.
The Calico daemonset and controller are currently Running, but several old token-refresh jobs remain stuck. The affected Valkey pod is scheduled on worker-2, has Calico annotations but no assigned pod IP, and its describe output has no Events. The IP pool shows roughly 128 addresses in use out of about 65,000, so exhaustion seems unlikely. I’ll compare the node’s CNI files, Calico logs, and kubelet journal with a healthy worker.
The machines are bare metal rather than cloud instances, so automatic replacement is not available. All workers report Kubernetes v1.35.0. I’m increasingly convinced that a clean rebuild with a supported Calico version and proper maintenance procedures would be safer than paying for another temporary restart job.