Fix: 429: Rate Limit Exceeded (GCP)

A 429 from Google Cloud means the service accepted your credentials, understood the request, and refused it anyway because you are sending more than a quota or rate limit allows. It shows up under names like rateLimitExceeded, userRateLimitExceeded, or RESOURCE_EXHAUSTED in gRPC APIs. The immediate handling is always the same (back off and retry), but the durable fix depends on which specific limit you hit, and GCP has three different kinds: per project API quotas, per user limits, and per resource rate limits.

429: Rate Limit Exceeded (GCP)

What this error means

Every GCP API enforces quotas: requests per minute per project, per user, and per region, plus service specific limits like write rates to a single Cloud Storage object or a Firestore document range. When your traffic crosses one of them, the API returns HTTP 429 (or gRPC RESOURCE_EXHAUSTED) instead of serving the call. This is throttling, not failure: the request was valid and will succeed once you are back under the limit. The error body usually carries the reason and, on newer APIs, structured QuotaFailure details naming the exact metric that tripped.

Common causes

• A batch job, backfill, or newly scaled service pushing a per minute API quota past its ceiling.
• Retry storms: clients retrying instantly without backoff, multiplying a small spike into a sustained overload.
• Per user limits when all traffic flows through one service account, concentrating what the API sees as a single caller.
• Per resource hot spots: repeated writes to one Cloud Storage object, one Firestore document range, or one Spanner key.
• Default quotas on a new project or newly enabled API that were never raised to match production traffic.
• Many jobs or teams sharing one project, so combined traffic hits project level quotas nobody individually expects.

How to fix it

STABILIZE first. Slow the source: pause or throttle the batch job, and make sure clients back off instead of hammering retries, so the traffic that matters keeps flowing under the limit.

1. DIAGNOSE which quota tripped. Find the 429s in the logs, including the service and reason:

gcloud logging read 'protoPayload.status.code=8 OR httpRequest.status=429' \
  --freshness 1h \
  --limit 20 \
  --format json

2. Check current quota usage for the service to see which metric is pinned at its limit:

gcloud services list --enabled --filter "storage"

gcloud alpha services quota list \
  --service storage.googleapis.com \
  --consumer projects/my-project \
  --format json

The Quotas page in the console (IAM and Admin, Quotas) shows the same data with current usage percentages, which is the fastest way to spot the metric at 100 percent.

3. FIX the client behavior. Confirm every caller uses exponential backoff with jitter and honors Retry-After. The official client libraries do; custom HTTP calls often do not. Cap concurrency in batch workers so offered load stays below the quota rather than relying on retries.

4. If the limit is legitimate growth, request a quota increase on the exact metric from the Quotas page or via the API:

gcloud alpha services quota update \
  --service compute.googleapis.com \
  --consumer projects/my-project \
  --metric compute.googleapis.com/read_requests \
  --unit 1/min/{project} \
  --value 3000

Increases above default ceilings go through Google review and can take days, so file them before the launch, not during it.

5. If the limit is per resource, redesign the access pattern: spread writes across many objects or keys, batch mutations, cache reads, or move high frequency counters to a datastore built for them.

How to prevent it

• Set alerts on quota usage (Monitoring has per quota metrics) so you see 80 percent before you see 429.
• Request quota increases as part of launch planning for any new high volume workload.
• Keep exponential backoff with jitter in every client, and test batch jobs at production concurrency in staging.
• Separate noisy batch workloads into their own projects so they cannot starve production APIs of shared quota.
• Avoid single hot objects and keys by design: shard counters, distribute prefixes, batch writes.

When to call a senior engineer

Call for help when 429s persist even though the quota page shows headroom, when a launch deadline collides with a quota increase stuck in review, or when the throttled workload cannot simply be slowed because the business depends on its throughput. Erzon engineers can pinpoint the exact limiting metric or hot resource from logs, restructure the access pattern to fit real quotas, and set up the monitoring that keeps the next spike from becoming an outage.