Fix: ThrottlingException: Rate exceeded (AWS)

You are getting ThrottlingException: Rate exceeded (or RequestLimitExceeded on EC2) back from an AWS API call, and part of your workload has stalled or is failing intermittently. This is a signal, not a hard failure: AWS is telling you that you are calling an API faster than your account is allowed to in that region right now. The good news is that the error is retryable, and once you understand which caller is hot you can usually make it disappear.

What this error means

Every AWS API has rate limits, applied per account and per region. These are separate from resource quotas (like the number of EC2 instances you can run) and cover how many requests per second you may send to a given API operation. When you cross the allowed request rate, the service rejects the call with ThrottlingException: Rate exceeded. EC2 uses the older name RequestLimitExceeded for the same idea.

This shows up across many services: EC2 DescribeInstances, CloudFormation stack operations, DynamoDB control plane calls (CreateTable, DescribeTable), STS AssumeRole, Auto Scaling, and Lambda control plane APIs among others. The key property is that the error is transient and retryable. A single call that was throttled will usually succeed a moment later, once your request rate drops back under the limit.

Common causes

• A tight polling loop that calls a describe or list API every few milliseconds instead of on a sensible interval.
• A describe or read call sitting inside a hot request path, so it fires once per user request rather than being cached.
• Many Lambda functions fanning out and calling the same control plane API at the same instant.
• Infrastructure tooling (Terraform or the AWS CDK) running with high parallelism, issuing dozens of describe calls at once.
• Cron jobs or scheduled tasks that all fire on the same minute across many hosts.
• Genuine growth: your legitimate demand has outgrown the default quota for that API.

How to fix it

STABILIZE

1. Enable the SDK retry behavior so calls back off automatically instead of failing hard. For the CLI and most SDKs, set the retry mode and attempt cap through environment variables:

export AWS_RETRY_MODE=adaptive
export AWS_MAX_ATTEMPTS=10

2. In application code, configure retries explicitly. In Python with boto3, use a botocore Config with adaptive mode:

import boto3
from botocore.config import Config

cfg = Config(
    retries={"max_attempts": 10, "mode": "adaptive"},
)

ec2 = boto3.client("ec2", config=cfg)
response = ec2.describe_instances()

The adaptive mode adds client side rate limiting on top of exponential backoff with full jitter, which is what you want while a workload is actively being throttled. If you prefer more predictable behavior, standard mode also applies exponential backoff plus jitter without the extra client side throttling.

3. Reduce concurrency at the source. Lower the number of parallel workers, threads, or Lambda invocations hitting the API until the errors stop.

DIAGNOSE

1. Find which API and which caller is hot. Query CloudTrail for the throttled operation:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=DescribeInstances \
  --max-results 50

2. Look at the source identity and frequency in the results. A single principal calling the same operation many times per second is your culprit. Watch for describe or list calls that repeat far more often than the underlying data changes.

3. Check your current limits in the Service Quotas console (or CLI) for the service in question, so you know how far over the line you are:

aws service-quotas list-service-quotas --service-code ec2

4. If Terraform or the CDK is involved, check whether it is running with high parallelism, which defaults to 10 concurrent operations in Terraform.

FIX

1. Cache describe and read results. If instance metadata or a table description changes rarely, fetch it once and reuse it rather than calling on every request.
2. Batch and paginate sensibly. Use batch APIs where they exist, and page through results with a reasonable page size instead of many small calls.
3. Replace polling with event driven patterns. Use EventBridge, SNS, or SQS notifications instead of looping on a describe call.
4. Spread scheduled work so cron jobs do not all fire on the same minute across hosts.
5. Lower Terraform parallelism when a plan or apply triggers throttling:

terraform apply -parallelism=2

6. If demand is legitimate, request a quota increase through Service Quotas for the specific operation.

How to prevent it

• Use rate aware clients everywhere: keep adaptive or standard retry mode on by default.
• Cache read heavy describe and list results with a sensible time to live.
• Prefer event driven notifications over polling loops.
• Monitor your usage against quotas and alarm before you reach the limit.
• Keep infrastructure tooling parallelism modest in shared accounts.

When to call a senior engineer

If throttling is causing customer facing failures during an incident, if you cannot identify which caller is hot from CloudTrail, or if a quota increase has been requested but the errors persist, bring in a senior engineer. The same applies when the throttled API sits on a critical path (deploys, autoscaling, failover) and you need the call pattern redesigned quickly and safely. Erzon can join mid incident, find the hot caller, and stabilize the workload with you.