Fix: Readiness probe failed: HTTP probe failed with statuscode: 503

This event means the kubelet called your container’s HTTP readiness endpoint and the app answered 503 Service Unavailable. Kubernetes did exactly what it is designed to do: it marked the pod NotReady and removed it from Service load balancing. The container is running, but it is telling Kubernetes it cannot serve traffic. If enough pods say that at once, your Service has no endpoints and users see errors.

Readiness probe failed: HTTP probe failed with statuscode: 503

What this error means

A readiness probe is Kubernetes asking your application “can you take traffic right now?” on a schedule. Unlike a liveness probe, failing it never restarts the container; it only controls whether the pod’s IP appears in the Service’s endpoints. A 503 is not the kubelet’s opinion, it is your application’s own health handler declaring itself unavailable. That makes this error different from a connection refused probe failure: the process is up and answering HTTP, and something inside it, a dependency check, a warm-up phase, or an overload condition, is returning the failure.

Common causes

• The health endpoint performs deep dependency checks (database, cache, downstream API) and one of those dependencies is down or slow.
• The app is still starting: migrations, cache warming, or JIT warm-up, and the probe begins before initialization completes.
• Overload: request queues are full or worker threads are saturated, so the health handler itself returns 503 or times out.
• A graceful shutdown in progress: the app flips to 503 while draining, and you are watching normal rollout behavior.
• Probe misconfiguration: wrong path or port, an endpoint that requires auth, or a failureThreshold and periodSeconds combination too aggressive for reality.

How to fix it

STABILIZE first. Check whether the Service still has ready endpoints. If every pod is NotReady because a shared dependency check fails, restoring that dependency (or temporarily pointing readiness at a shallow endpoint) brings traffic back immediately.

kubectl get endpoints <service-name>
kubectl get pods -l app=myapp -o wide

1. DIAGNOSE by asking the app the same question the kubelet asks. Exec into the pod (or a debug container) and hit the probe path directly, then read the app logs from the same time window.

kubectl describe pod <pod-name> | grep -A3 Readiness
kubectl exec <pod-name> -- wget -qO- -S http://127.0.0.1:8080/healthz
kubectl logs <pod-name> --since=10m | tail -50

2. If the 503 comes from a dependency check, fix the dependency, and reconsider whether readiness should depend on it at all. A shared database blip should not remove every pod from the Service.

3. If the pod fails readiness only at startup, FIX the probe timing with a startup probe instead of guessing at delays:

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 5
readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 10
  failureThreshold: 3

4. If overload causes the 503s, readiness removal makes it worse: fewer ready pods receive the same traffic and fail in turn. Scale out first, then fix the capacity or the slow code path.

kubectl scale deployment/myapp --replicas=6

5. Verify the fix by watching the pod return to Ready and the endpoints repopulate.

kubectl get pods -l app=myapp -w

How to prevent it

• Keep readiness checks shallow: the process, its port, and resources it exclusively owns. Handle shared dependency failures in application logic.
• Use a startupProbe for slow-starting services rather than inflated initial delays.
• Alert on the ready endpoint count per Service, not only on pod restarts, so a silent readiness drop is visible.
• Load test the health endpoint too: it must answer quickly under peak traffic, not compete for the same saturated worker pool.
• During rollouts, set sane maxUnavailable values so a readiness regression cannot drain the whole Service.

When to call a senior engineer

Call for help when readiness flaps under load and every scaling change just moves the problem, when a rollout drains all endpoints and the cause is buried in the app’s health logic, or when nobody can say what the health endpoint actually checks. Erzon engineers can trace the 503 from kubelet to handler, redesign the probe strategy so partial failures stay partial, and leave your Services with endpoint behavior you can trust during incidents.