Fix: Quota exceeded, rateLimitExceeded (HTTP 403)

A Google Cloud API call failed with HTTP 403 and a message starting with Quota exceeded for quota metric. The reason field at the end tells you which of two very different problems you have. rateLimitExceeded means you sent requests faster than a per-minute quota allows, and the call will succeed if you slow down and retry. quotaExceeded on a resource operation usually means an allocation quota, a hard cap on how much of something the project can hold, such as CPUs in a region, and no amount of retrying will fix it.

What this error means

Google Cloud enforces two kinds of quotas on every project. Rate quotas limit how many API calls you can make per minute, per user, or per region, and they reset on a sliding window. Allocation quotas limit how much of a resource the project can hold at once, such as CPUS per region, in-use IP addresses, or persistent disk terabytes, and they only free up when you delete resources or get the limit raised.

The error message names the exact quota metric, the limit that was hit, the service, and the consumer project number. Read those fields, they are the diagnosis. A limit named Queries per minute or Read requests per minute per user is a rate quota. A metric like CPUS, IN_USE_ADDRESSES, or SSD_TOTAL_GB on a Compute operation is an allocation quota, and Compute typically surfaces it as QUOTA_EXCEEDED on the operation rather than a plain API 403.

Also check which project is the consumer. When you call an API through a service account or an API key, the quota is charged to the quota project associated with the credentials, which is not always the project that owns the resource. A surprise 403 from a project number you do not recognize usually means the quota project on your client credentials is wrong.

Common causes

• A burst of parallel API calls, often from a deploy script, a controller reconcile loop, or a data pipeline fanning out workers, tripping a per-minute rate quota.
• A retry loop without backoff turning one throttled call into a storm that keeps the quota pinned.
• An autoscaler or Terraform apply trying to create instances past the regional CPUS or address allocation quota.
• A new project or a project on the free tier, which starts with much lower default quotas than a mature project.
• Client credentials billing quota to the wrong quota project, so you exhaust a shared or default project’s limits instead of your own.
• Regional imbalance, where one region’s quota is exhausted while others sit idle.

How to fix it

STABILIZE. If this is a rate quota, get the callers to back off so legitimate traffic can get through. Every Google client library supports retry with exponential backoff, make sure it is on and that your own outer retry loops are not stacked on top of it. Cut worker concurrency, add jitter so schedulers do not fire in the same second, and pause any batch job that is hammering the API. If this is an allocation quota blocking a scale-up, free capacity immediately by deleting stopped instances, unattached disks, and reserved but unused static IPs in the affected region.

DIAGNOSE. First confirm exactly which quota is exhausted. For Compute allocation quotas, read current usage against limits per region:

gcloud compute regions describe us-central1 \
  --project=PROJECT_ID \
  --format="table(quotas.metric, quotas.usage, quotas.limit)"

gcloud compute project-info describe \
  --project=PROJECT_ID \
  --format="table(quotas.metric, quotas.usage, quotas.limit)"

For any service, the Cloud Quotas API lists every quota and its effective value:

gcloud quotas info list \
  --service=compute.googleapis.com \
  --project=PROJECT_ID

For rate quotas, check usage over time in Cloud Monitoring with the metrics serviceruntime.googleapis.com/quota/rate/net_usage and serviceruntime.googleapis.com/quota/allocation/usage, grouped by quota metric. Look at one minute resolution, because bursts that trip the limit are invisible on hourly charts. Identify who is calling: filter the metric by credential or check audit logs for the method being spammed.

If the consumer project number in the error is not the project you expected, inspect and fix the quota project on your credentials:

gcloud auth application-default set-quota-project PROJECT_ID

FIX. For a rate quota, the durable fix is client behavior plus, when genuinely needed, a higher limit. Batch calls where the API supports it, cache reads, and smooth bursts with a client-side rate limiter. Then request an increase either in the console under IAM & Admin, Quotas and system limits, or with the Cloud Quotas API:

gcloud quotas preferences create \
  --service=compute.googleapis.com \
  --quota-id=ReadRequestsPerMinutePerProject \
  --preferred-value=3000 \
  --project=PROJECT_ID \
  [email protected]

Use the quota ID shown by gcloud quotas info list for the metric in your error. For an allocation quota, either delete resources you no longer need, spread the workload across another region with headroom, or request a higher cap the same way. If the request is large, attach a short justification, it materially speeds up review.

How to prevent it

• Turn on the built-in retry and backoff in every Google client library, and never wrap it in a tight outer retry loop.
• Alert on quota usage above 80 percent using the serviceruntime quota metrics in Cloud Monitoring, for both rate and allocation quotas.
• Request quota increases ahead of launches and load tests, not during them, and record the approved values in your infra docs.
• Set the quota project explicitly on service credentials so consumption lands where you expect.
• Spread capacity across regions before a single region’s allocation quota becomes a single point of failure.
• Smooth scheduled jobs with jitter so hundreds of workers do not hit the same per-minute window together.

When to call a senior engineer

Call for help when a retry storm keeps re-pinning the quota faster than you can trace its source, when an autoscaler is blocked on allocation quota during an incident and you need capacity moved across regions safely, or when quota increase requests keep getting rejected and the architecture needs to change instead. Erzon engineers can trace the offending callers through audit logs, restructure the client layer with proper batching and backoff, and get the workload back under its limits without guesswork.