Fix: exceeded quota: requests.cpu, requested: ... limited: ...

This error means the Kubernetes API server refused to create a pod because the namespace’s ResourceQuota has no remaining room for the CPU the pod requests. The message shows the arithmetic: what the pod asked for, what is already used, and the hard limit. It is not a scheduling failure and no pod ever gets created, which is why this one is sneaky: Deployments quietly run below their replica count and the error hides in ReplicaSet events.

exceeded quota: requests.cpu, requested: ... limited: ...

What this error means

A ResourceQuota object caps the total resources all pods in a namespace can request, most commonly requests.cpu, requests.memory, and their limit counterparts. Every pod creation is checked against the quota at admission time: current usage plus the new pod’s requests must fit under the hard cap, or the API server rejects the request outright. The rejection happens before scheduling, so nothing appears in kubectl get pods. The controller that tried to create the pod (usually a ReplicaSet) records the failure as an event and retries, and it will keep failing until quota frees up or the quota grows.

Common causes

• The namespace is genuinely at its CPU request cap: normal growth, a scale-up, or an HPA increase pushed it over.
• Workloads over-request CPU: requests were set by copy-paste or guesswork and are far above real usage, wasting quota.
• A rollout temporarily needs headroom: surge pods during a Deployment update request quota on top of the old pods still running.
• Leftover workloads nobody remembers, completed Jobs, or orphaned ReplicaSets still holding quota.
• A quota sized long ago for a smaller version of the service, never revisited.

How to fix it

STABILIZE first. Confirm what is blocked and how much quota is left, so you know whether you are one pod short or completely wedged.

kubectl describe quota -n <namespace>
kubectl describe deployment <name> -n <namespace> | grep -A5 Conditions
kubectl get events -n <namespace> --sort-by=.lastTimestamp | grep -i quota

1. DIAGNOSE who is holding the quota. List every pod’s CPU requests and compare against what they actually use.

kubectl get pods -n <namespace> \
  -o custom-columns='NAME:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu'
kubectl top pods -n <namespace> --sort-by=cpu

2. Reclaim quota that is simply wasted. Delete finished Jobs and any abandoned workloads still counting against the namespace.

kubectl delete jobs -n <namespace> --field-selector=status.successful=1

3. FIX over-requesting workloads by right-sizing their requests to observed usage plus headroom. This is usually where most of the quota comes back:

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "1"
    memory: "512Mi"

4. If usage is honest and the namespace has genuinely outgrown its share, raise the quota deliberately, after confirming the cluster has node capacity behind it:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: myteam
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi

5. For rollouts that fail only during surge, reduce the temporary overhead so updates fit inside the quota:

strategy:
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 1

How to prevent it

• Alert on quota utilization per namespace (used versus hard) at around 80 percent, so teams see the wall before deploys hit it.
• Review requests against real usage regularly, or let a vertical pod autoscaler recommend them.
• Set a LimitRange with default requests so unconfigured pods get sane values instead of team-wide copy-paste numbers.
• Clean up completed Jobs automatically with ttlSecondsAfterFinished.
• Size quotas with rollout surge in mind, not just steady-state replicas.

When to call a senior engineer

Call for help when quota rejections block releases and nobody can say which workloads deserve the CPU they request, when HPA scale-ups and quotas fight each other during traffic peaks, or when cluster capacity planning has drifted so far that every namespace is squeezed. Erzon engineers can audit requests against real usage across the cluster, rebuild the quota and LimitRange scheme around measured numbers, and give you the alerting that ends surprise deploy failures.