Fix: ImagePullBackOff (Kubernetes)

ImagePullBackOff means the kubelet on the node tried to pull your container image, failed, and is now waiting an increasing amount of time between retries. The pod never starts, so if this is a fresh deploy or a scale-up, real capacity is missing. The status itself says nothing about why the pull failed. The actual reason, a bad tag, a 401 from the registry, or a network timeout, is in the pod events.

ImagePullBackOff

What this error means

Before a container can start, the node’s container runtime must have the image locally. The kubelet asks the runtime to pull it from the registry named in the image reference. When that pull fails, the pod briefly shows ErrImagePull, and after repeated failures Kubernetes applies an exponential backoff (up to five minutes between attempts) and reports ImagePullBackOff. The pod stays in this loop forever until the pull can succeed. Nothing is wrong with the pod spec scheduling or the node itself: the node simply cannot obtain the image.

Common causes

• The image tag does not exist: a typo, a CI pipeline that failed before pushing, or a tag that was deleted or never built for this architecture.
• Missing or wrong registry credentials: private registries return 401 or 403 and the pull fails with an authentication error.
• The imagePullSecrets entry is absent, in the wrong namespace, or contains an expired token (common with cloud registries that issue short-lived credentials).
• The node cannot reach the registry: DNS failure, egress firewall, proxy misconfiguration, or a registry outage.
• Registry rate limiting, which is frequent with anonymous pulls from Docker Hub on busy clusters.
• The image exists but not for the node’s CPU architecture, for example an amd64-only image on an arm64 node.

How to fix it

STABILIZE first. If a deploy caused this, roll back to the last image tag that pulled successfully so the workload keeps running while you investigate.

kubectl rollout undo deployment/myapp

1. Read the pod events. The last event names the exact pull failure: not found, unauthorized, or a network error.

kubectl describe pod <pod-name> | tail -20

2. DIAGNOSE based on the message. “manifest unknown” or “not found” means the tag does not exist: verify it in the registry and check what the pod is actually asking for.

kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].image}'

3. “unauthorized” or “authentication required” means credentials. Confirm the pull secret exists in the pod’s namespace and is referenced by the pod or its service account.

kubectl get secret regcred -n <namespace>
kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=deploy \
  --docker-password='<token>' \
  -n <namespace>

4. FIX the pod spec to reference the secret if it does not already:

spec:
  imagePullSecrets:
    - name: regcred
  containers:
    - name: myapp
      image: registry.example.com/myapp:v1.42.0

5. For network failures, test the registry from the node itself. If DNS or egress fails there, the fix is in networking, not Kubernetes.

kubectl debug node/<node-name> -it --image=busybox -- \
  wget -S https://registry.example.com/v2/

Once the cause is fixed, delete the stuck pod (or let the Deployment replace it) and the next pull attempt runs immediately instead of waiting out the backoff.

How to prevent it

• Deploy immutable, unique tags (or digests) written by CI after a successful push, never a manually typed tag.
• Make credential renewal automatic for registries that issue expiring tokens, and attach pull secrets to the namespace’s default service account.
• Add a CI gate that pulls the image before applying the manifest, so a missing tag fails the pipeline instead of production.
• Authenticate all pulls from Docker Hub or mirror hot images to your own registry to avoid rate limits.
• Publish multi-arch images if your cluster mixes amd64 and arm64 nodes.

When to call a senior engineer

Call for help when pulls fail intermittently across nodes with no consistent error, when private registry authentication works everywhere except inside the cluster, or when an air-gapped or proxied environment makes every image pull a fight. Erzon engineers can trace the pull path from kubelet to registry, fix the credential and network layer, and set up an image delivery pipeline that does not strand pods in backoff again.