Fix: 503: The service is currently unavailable (GCP)

A 503 in Google Cloud is deliberately generic: the service handling your request, or something on the path to it, could not serve it right now. It appears when a Google API has a transient backend failure, when your external HTTP load balancer has no healthy backend to route to, and when your own Cloud Run or GKE workload is overloaded. Debugging it efficiently is a matter of attribution first: decide whose 503 it is, then apply the fix for that layer instead of guessing across all of them.

503: The service is currently unavailable (GCP)

What this error means

HTTP 503 maps to gRPC UNAVAILABLE, and Google’s own guidance treats it as a retryable, usually transient condition. Three distinct layers can generate it. Google’s APIs return it during brief internal failures or maintenance, and well behaved clients absorb these with retries. The external HTTP load balancer returns it when it cannot pick a backend, with the reason recorded in the request log’s statusDetails field. And your own services return it when they are saturated or their dependencies are down. The same three characters, three different owners, three different fixes.

Common causes

• A transient Google side blip on an API call, visible as sporadic single failures that a retry absorbs.
• An actual Google Cloud incident in your region, confirmed on the Service Health dashboard.
• Your load balancer has no healthy backend: health checks fail because the app is down, the check is misconfigured, or firewall rules block Google’s probe ranges (35.191.0.0/16 and 130.211.0.0/22).
• Backend saturation: instances at max utilization or a Cloud Run service at max instances, so requests are shed.
• Empty backends after a bad deploy, an autoscaler stuck at zero, or GKE pods crash looping behind a NEG.
• Retry storms from your own clients amplifying a brief failure into a sustained one.

How to fix it

STABILIZE first. If clients fail hard on every request, confirm retries with backoff are active for idempotent calls, and if a recent deploy correlates with the onset, roll it back now.

1. Attribute the 503. Check for an active Google incident, then find who generated the response:

gcloud beta service-health events list --project my-project

gcloud logging read 'httpRequest.status=503' \
  --freshness 1h \
  --limit 20 \
  --format 'value(resource.type, jsonPayload.statusDetails, httpRequest.requestUrl)'

A resource.type of http_load_balancer with statusDetails like failed_to_pick_backend or backend_connection_closed_before_data_sent_to_client points at your backends. Errors on API calls from your code with no load balancer involvement point at the Google API path.

2. DIAGNOSE backend health if the load balancer generated it:

gcloud compute backend-services get-health my-backend-service \
  --global \
  --format json

Every backend UNHEALTHY means the app is down, the health check is wrong, or probes are blocked. Verify the firewall rule that admits Google’s health check ranges:

gcloud compute firewall-rules create allow-health-checks \
  --network default \
  --source-ranges 35.191.0.0/16,130.211.0.0/22 \
  --allow tcp:8080

3. FIX the backend layer. If the app crashed, restart or roll back and read its logs for the boot failure. If health checks are misconfigured, align the check port and path with what the app actually serves. If the group is empty, fix the autoscaler minimum or the deployment that removed everything.

4. If backends are healthy but saturated, add capacity: raise max instances on Cloud Run, raise autoscaler ceilings on instance groups, or scale the GKE deployment. Load shedding 503s disappear when utilization drops below the target.

5. If it is a confirmed Google incident, do not redeploy anything. Keep retries with backoff on, fail over to another region if your architecture supports it, and track the incident to resolution.

How to prevent it

• Keep exponential backoff with jitter on every API client, and idempotency keys on non idempotent operations.
• Alarm on backend health and on load balancer 5xx rates by statusDetails, so attribution is instant during an incident.
• Run backends in more than one zone, with autoscaler minimums that survive a zone loss.
• Keep the firewall rules for Google health check ranges managed as code so nobody deletes them.
• Load test to find the saturation point before production traffic does, and set scaling limits above it.

When to call a senior engineer

Call for help when 503s persist and attribution keeps bouncing between the load balancer, the app, and Google, when health checks pass but users still get errors, or when scaling up somehow makes it worse. Erzon engineers can read the load balancer logs and backend telemetry to pin the failing layer with evidence, restore healthy serving, and leave you with the dashboards that make the next 503 a five minute diagnosis.