Fix: 429 Too Many Requests

A 429 Too Many Requests means some rate limiter between the client and the origin decided the caller exceeded its allowance. It is not an outage and not a bug in the request itself: the same request would succeed if it arrived slower. The work is identifying which limiter fired, respecting its retry signal, and reshaping the traffic so it stops firing.

429 Too Many Requests

What this error means

Rate limiting is enforced at many layers: third-party APIs, API gateways, CDNs and WAFs, reverse proxies, and application middleware. Each tracks requests against a key (an API key, a token, an IP, a user ID) over a window, and answers 429 when the bucket is empty. A well-behaved limiter includes a Retry-After header saying when to try again, and often X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset (or their RateLimit-* standardized equivalents) so clients can pace themselves before hitting zero.

The critical detail is that limits are usually about burst shape, not just volume. A client that is far under its hourly quota can still trip a per-second or per-minute burst limit by sending requests in a tight loop.

Common causes

• A retry loop without backoff: a failed call retried in a hot loop converts one error into a limit violation.
• Bursty batch jobs: a cron task firing hundreds of API calls in the first second of the minute.
• Fan-out amplification: one user action triggering many downstream API calls per request.
• Shared limit keys: all servers egressing via one NAT IP, or one API key shared across services.
• Missing caching, so the same resource is fetched repeatedly instead of once.
• A WAF or bot-protection rule classifying legitimate traffic as abusive.

How to fix it

STABILIZE first. If a background job or retry loop is generating the flood, pause it. That immediately stops feeding the limiter, and most limits reset within seconds to minutes once the pressure stops.

1. Capture a failing response with full headers to see which layer answered and what it told you.

curl -sS -D - -o /dev/null https://api.example.com/v1/resource \
  -H "Authorization: Bearer $TOKEN"

Look for Retry-After and the X-RateLimit-* family, and note the response body shape, which usually identifies the gateway or provider.

2. Find who is burning the budget. Group your outbound calls (or inbound, if you serve the 429s) by caller and endpoint over the incident window.

awk '$9 == 429 {print $1, $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head

DIAGNOSE from the pattern. One caller dominating the count is a runaway client or retry loop. Many callers hitting one endpoint points at a fan-out or caching gap. If the 429 comes from a third-party API, compare your observed rate to their documented limit and check whether the key is shared more widely than you thought.

FIX on the client side with backoff plus jitter, honoring Retry-After when present:

import random, time
import requests

def get_with_backoff(url, headers, max_attempts=6):
    for attempt in range(max_attempts):
        resp = requests.get(url, headers=headers, timeout=10)
        if resp.status_code != 429:
            return resp
        retry_after = resp.headers.get("Retry-After")
        if retry_after is not None:
            delay = float(retry_after)
        else:
            delay = min(60, (2 ** attempt)) * random.uniform(0.5, 1.5)
        time.sleep(delay)
    resp.raise_for_status()

Then reshape the traffic so the limiter stops firing at all: smooth bursts with a client-side token bucket or a queue, cache responses that do not change per call, and batch endpoints that support it. If you own the limiter and legitimate traffic is being throttled, raise the burst allowance rather than the total rate; in Nginx that is the burst parameter on limit_req.

How to prevent it

• Every external API call goes through a client wrapper with backoff, jitter, and Retry-After handling built in.
• Track X-RateLimit-Remaining and pace proactively instead of running the bucket to zero.
• Spread scheduled jobs across the window instead of firing at second zero of the minute.
• Cache and deduplicate reads; the cheapest request is the one you do not send.
• Give each service its own API key so one noisy consumer cannot starve the rest, and so the source of a flood is obvious.

When to call a senior engineer

Call for help when 429s persist after adding backoff (which usually means a shared key, a NAT bottleneck, or a WAF rule you have not identified), when a third-party quota genuinely cannot cover your workload and the integration needs redesign around caching, batching, or webhooks, or when your own rate limiting is throttling paying customers. Erzon engineers can trace exactly which limiter fires and why, instrument your consumption against every quota, and rebuild the traffic shape so the limits stop being the story.