Fix: ErrImagePull (Kubernetes)

ErrImagePull is the immediate error state when the kubelet asks the container runtime to pull your image and the pull fails. The container cannot start without its image, so the pod sits in Pending or Waiting. Left alone it becomes ImagePullBackOff as retries back off, but the cause is already visible right now in the pod events, and reading that one Failed event is the whole diagnosis.

ErrImagePull

What this error means

Every container in a pod names an image reference: registry host, repository, and tag or digest. On scheduling, the node’s runtime resolves and downloads that image unless it is already cached locally. ErrImagePull means the most recent pull attempt returned an error. The error string attached to the event is specific: “manifest unknown” means the tag or repository does not exist, “unauthorized” means the registry rejected the credentials, and i/o timeouts or DNS errors mean the node could not reach the registry at all. The pod spec is accepted and scheduled; only the image delivery is broken.

Common causes

• A wrong or nonexistent image reference: mistyped repository, a tag CI never pushed, or a deleted tag.
• Private registry authentication failing: no imagePullSecrets, a secret in the wrong namespace, or an expired token.
• Registry unreachable from the node: DNS, egress firewall rules, a required proxy, or a registry outage.
• Rate limiting on anonymous pulls, especially from Docker Hub.
• Architecture mismatch: the manifest exists but has no image for the node’s CPU architecture.
• A very large image timing out on slow node networks before the download completes.

How to fix it

STABILIZE first. If a new rollout introduced the broken reference, roll back so the previous ReplicaSet with a pullable image serves traffic while you fix the pipeline.

kubectl rollout undo deployment/myapp
kubectl rollout status deployment/myapp

1. Read the exact pull error from the events.

kubectl get events --field-selector involvedObject.name=<pod-name> \
  --sort-by=.lastTimestamp

2. DIAGNOSE from the message. For “manifest unknown” or “not found”, print the reference the pod is really using and verify it exists in the registry.

kubectl get pod <pod-name> \
  -o jsonpath='{range .spec.containers[*]}{.image}{"\n"}{end}'
docker pull registry.example.com/myapp:v1.42.0

3. For “unauthorized”, inspect the pull secret and confirm the registry host inside it matches the image reference exactly.

kubectl get secret regcred -n <namespace> \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d

4. FIX the broken piece. Recreate the secret with current credentials if the token expired, and reference it in the workload:

spec:
  template:
    spec:
      imagePullSecrets:
        - name: regcred
      containers:
        - name: myapp
          image: registry.example.com/myapp:v1.42.0

5. For network errors, verify the node can resolve and reach the registry. A failure here means the fix belongs in DNS, firewall, or proxy configuration.

kubectl debug node/<node-name> -it --image=busybox -- \
  nslookup registry.example.com

After fixing the cause, a new pull attempt happens on the normal retry, or immediately if you delete the pod and let the controller recreate it.

How to prevent it

• Let CI generate and push the tag, then inject that exact tag into the manifest, so a human never types an image reference.
• Prefer pinning by digest for critical workloads so the deployed artifact cannot change or vanish under a mutable tag.
• Automate rotation of registry credentials that expire, and alert on pull failures cluster-wide, not per app.
• Keep a pull-through cache or mirror inside your network for external base images.
• Build and push multi-arch manifests if your node pools mix architectures.

When to call a senior engineer

Call for help when image pulls fail only in one cluster or one node pool and nothing in the manifests explains it, when registry authentication involves cloud IAM chains nobody fully owns, or when deploys are blocked and rollback is not an option. Erzon engineers can walk the pull path end to end, from pod spec through credentials, DNS, and egress to the registry, restore deploys quickly, and harden the pipeline so a bad reference never reaches the cluster again.