Fix: 503 Service Temporarily Unavailable (ALB: no healthy targets)

A 503 from an Application Load Balancer means the ALB itself generated the response because it had no healthy target to forward the request to. Your load balancer is fine; every instance, container, or IP behind it is either failing health checks or missing from the target group. The good news is that the failure is measurable: each target carries a health reason code that says exactly why the ALB pulled it.

503 Service Temporarily Unavailable (ALB: no healthy targets)

What this error means

The ALB continuously probes every registered target on the configured health check path, port, and protocol. A target that fails the check more times than the unhealthy threshold is taken out of rotation. When that happens to every target in the group, the ALB has nowhere to route and answers requests itself with 503. There are three distinct states behind it: the group has targets and all are unhealthy (health checks fail), the group is empty (registration or deployment removed everything), or targets are still in initial state after registration and none has passed the healthy threshold yet.

Common causes

• Health check misconfiguration: wrong path, wrong port, or a success code mismatch (the endpoint returns 301, 302, or 401 while the check requires 200).
• Security groups: the target’s security group does not allow the health check traffic from the ALB’s security group.
• The application is down or crash looping on every instance after a bad deploy.
• The application is up but too slow: response time exceeds the health check timeout, so checks fail under load and targets flap.
• The target group is empty because an Auto Scaling group scaled to zero, a deployment deregistered everything, or ECS tasks keep failing to start.
• The health check endpoint depends on a downstream (database, cache) that is itself down, so all targets fail together even though the processes run.

How to fix it

STABILIZE first. One healthy target restores service. Find the fastest path to one: roll back the deploy that broke the app, or if the health check itself is misconfigured, correct it so already working targets pass.

1. DIAGNOSE with the target health reasons, which name the exact failure per target:

aws elbv2 describe-target-health \
  --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/app/abc123 \
  --query 'TargetHealthDescriptions[].{Id:Target.Id,State:TargetHealth.State,Reason:TargetHealth.Reason,Desc:TargetHealth.Description}'

Target.ResponseCodeMismatch means the endpoint answers with the wrong status. Target.Timeout means the check gets no answer in time, pointing at overload or a security group. Target.FailedHealthChecks with connection refused means nothing listens on the check port. An empty result means the group has no registered targets at all.

2. Compare the configured check against reality:

aws elbv2 describe-target-groups \
  --target-group-arns arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/app/abc123 \
  --query 'TargetGroups[0].{Path:HealthCheckPath,Port:HealthCheckPort,Matcher:Matcher.HttpCode,Timeout:HealthCheckTimeoutSeconds,Interval:HealthCheckIntervalSeconds}'

Then reproduce the exact check from a host inside the VPC:

curl -i http://10.0.1.25:8080/healthz

3. FIX according to what you found. If the endpoint returns a redirect or auth challenge, either point the check at a path that returns 200 or widen the matcher:

aws elbv2 modify-target-group \
  --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/app/abc123 \
  --health-check-path /healthz \
  --matcher HttpCode=200-299

4. If checks time out, verify the target security group allows inbound from the ALB security group on the check port, then look at application latency and capacity. If the group is empty, fix the supply side: check Auto Scaling activity or ECS service events to see why instances or tasks are not registering.

5. If the app is crash looping after a deploy, roll back first and debug the release afterward. The ALB cannot help until at least one process stays up.

How to prevent it

• Give the app a dedicated lightweight health endpoint that verifies the process is serving but does not fan out to every dependency.
• Alarm on the target group’s HealthyHostCount dropping, not just on 5xx rates, so you see targets draining before the last one goes.
• Keep at least two targets in separate availability zones so a single instance failure never empties the group.
• Gate deployments on target health: new instances must pass checks before old ones are deregistered.
• Load test with health check behavior in mind, since a timeout tuned for idle traffic will flap under peak load.

When to call a senior engineer

Call for help when targets flap under load and every capacity guess makes it worse, when ECS tasks pass local checks but never reach healthy in the group, or when a multi service outage keeps tripping health checks through shared dependencies. Erzon engineers can correlate target health reasons with application and infrastructure metrics, find the real constraint, and rebuild the health check and deployment flow so one bad release cannot empty your target group again.