Our cluster is required to pull images through an internal buffered registry, which fetches them from an upstream location. That buffered registry went down yesterday, preventing some pods from starting because they could not pull their images. Is there a way to configure a pod or deployment with a secondary image location, or otherwise allow image pulls to fall back to another registry?
3 Answers
If the image already exists somewhere in the cluster but pods are being scheduled onto different nodes, a peer-to-peer image cache such as Spegel can help distribute images between nodes. It still makes sense to fix the underlying registry availability, since new images and completely fresh nodes will need a reliable source.
Check the imagePullPolicy carefully. If it is IfNotPresent, a node can start the pod from its local image cache without contacting the registry. With Always, the kubelet generally contacts the registry to resolve the image, so an outage can still prevent startup even when layers are cached. This is not a substitute for a highly available registry, but it can reduce failures when images are already present.
A single Kubernetes image field does not support a list of registries or automatic fallback locations. The usual solution is to make the registry highly available, with redundant instances or a managed service. You could also use separate deployments pointing at different registries and switch traffic between them with a blue-green setup.
The pods may have been moved between nodes, which caused short restarts. The pull policy appears to be Always, so I’m wondering whether Kubernetes will still use a cached local image when the registry is unavailable.