Fix: CrashLoopBackOff (Kubernetes)

CrashLoopBackOff means a container in your pod started, exited, and Kubernetes has restarted it repeatedly, adding a growing delay between attempts. The important thing to understand up front is that CrashLoopBackOff is not the cause of anything. It is the symptom. The container is exiting for some reason, and the kubelet is backing off before each retry, so what you actually need is the reason the container exits. This guide walks the path from the loop back to the underlying crash.

What this error means

When a container exits, the kubelet restarts it according to the pod restart policy. If it keeps exiting quickly, the kubelet slows down the retries, delaying up to five minutes between attempts, and reports the pod as CrashLoopBackOff. The container is running your process, that process is stopping (cleanly or by being killed), and the loop is Kubernetes doing exactly what you told it to do. Your job is to read the exit code and the last output to learn why the process stopped.

Common causes

• Application error on startup: an unhandled exception because a required environment variable or secret is missing, a config file is malformed, or a dependency such as the database or a cache is unreachable.
• Exit code 137: the container was OOMKilled because it exceeded its memory limit. The limit is too low, or the app leaks or spikes memory on start.
• A failing liveness probe: the app is healthy but slow to start, the probe fails before it is ready, and Kubernetes kills and restarts a container that was never actually broken.
• A misconfigured command or entrypoint: exit code 127 means a command was not found, and exit code 126 means a file was found but is not executable.
• A migration, seed, or config step that fails on boot and exits non zero every time.
• An image that expects a volume, secret, or config map to be mounted, and that mount is not present, so the process fails immediately.

How to fix it

Stabilize

1. Confirm the state and the restart count so you know how fast it is looping.

kubectl get pods -n YOUR_NAMESPACE

2. If this is taking down other traffic, scale the deployment to zero to stop the churn while you investigate, then scale back after the fix.

kubectl scale deployment YOUR_DEPLOYMENT --replicas=0 -n YOUR_NAMESPACE

Diagnose

1. Read the events and the last state. The Last State block shows the exit code and reason, which usually names the cause outright.

kubectl describe pod YOUR_POD -n YOUR_NAMESPACE

2. Read the output of the container that crashed, not the fresh one. The --previous flag is the key step most people skip.

kubectl logs YOUR_POD -c YOUR_CONTAINER --previous -n YOUR_NAMESPACE

3. If the container dies too fast to log, override its command so it stays alive, then exec in and inspect config, files, and connectivity from inside.

kubectl debug YOUR_POD -it --image=busybox --target=YOUR_CONTAINER -n YOUR_NAMESPACE

Fix

Apply the fix that matches the exit code and logs. For missing config, add the correct environment variable or secret reference. For exit code 137, raise the memory limit:

resources:
  requests:
    memory: "256Mi"
  limits:
    memory: "512Mi"

For a slow starter that a liveness probe is killing, give it room to boot with a startupProbe and a realistic initial delay:

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 10
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 10

For a bad command or missing file, correct the command and args. For a dependency that must exist before the app starts, add an init container that waits for it rather than crashing the main container. Then roll out and watch the pod come up clean.

How to prevent it

• Separate liveness and readiness. Readiness gates traffic, liveness restarts a truly stuck process. Do not use liveness to check dependencies.
• Add a startupProbe for anything with a slow boot, so the liveness probe never fires during startup.
• Set memory requests and limits from observed usage, and revisit them after load changes.
• Validate config and required environment variables at startup with a clear error message, and fail fast with intent rather than a stack trace.
• Keep migrations and one time setup in jobs or init containers, not in the main container entrypoint.

When to call a senior engineer

If the exit code and previous logs do not point to an obvious cause, if the crash only happens under production load or on specific nodes, or if raising limits and fixing probes just moves the failure somewhere else, the loop is hiding a deeper issue such as a memory leak, a race on a shared dependency, or a bad interaction between the scheduler and your resource settings. That is the point to bring in someone who can read the whole system rather than one pod, and Erzon can jump into a live incident and work it with you.