Our cluster is required to pull images through an internal buffered registry, which then retrieves them from an upstream location. That buffered registry was unavailable yesterday, and some pods could not start because they were scheduled on nodes that needed to pull their images. Is there a way to configure a second image location or fallback registry for the same container image?
2 Answers
If the image already exists somewhere in the cluster, a node-to-node image-sharing tool such as Spegel can reduce dependence on the central registry. That helps with cached images, but it is not a true fallback for images that no node has available, so the registry itself should still be made redundant.
A container image's `image` field is a single reference, not a list of registries that Kubernetes can try in sequence. The better fix is to make the registry endpoint highly available, with redundant backends or a reliable mirror. If the image is already cached on a node, `IfNotPresent` may let the pod start without pulling it again. With `Always`, Kubernetes still checks the registry when starting a container, so an outage can prevent startup even when layers are cached. You can also consider pre-pulling images or using separate registries with a deployment strategy that switches traffic between healthy versions.
The pods may have been moved between nodes, so they could have needed to start again on machines without the image cached. Their pull policy appears to be `Always`, which probably explains why a registry outage caused the startup failures.