Fix: OOMKilled (Exit Code 137)

A Kubernetes container killed with OOMKilled and exit code 137 hit its memory limit, and the kernel terminated it. This is one of the most common production failures in Kubernetes, and the good news is that it is usually diagnosable in a few minutes. The hard part is telling apart a limit that is simply set too low from a genuine memory leak, because the fix is different in each case.

What this error means

Exit code 137 is 128 plus signal 9 (SIGKILL). The container exceeded its cgroup memory limit, the Linux OOM killer selected the process, and it was killed immediately without a chance to clean up. In the pod status you will see State: Terminated, Reason: OOMKilled, Exit Code: 137.

There are two distinct scenarios that produce this. In the first, the container hit its own resources.limits.memory and was killed in isolation. In the second, the whole node ran low on memory and the kubelet evicted pods to reclaim it. Node level eviction shows a Reason of Evicted with a memory pressure message rather than OOMKilled on the container, so read the fields carefully. This page focuses on the container limit case.

Common causes

• The memory limit is set lower than the container’s normal working set, so ordinary load kills it.
• A genuine memory leak where usage climbs steadily until it crosses the limit.
• A runtime that does not respect cgroup limits, so the heap grows past the container limit (older JVMs, Node without an old space cap).
• A workload spike (large request, batch job, cache warm) that briefly needs more memory than the limit allows.
• No limit set at all, so the container is capped only by node memory and gets killed under node pressure.

How to fix it

STABILIZE first. If the service is down, buy headroom by raising the limit so pods stay up while you investigate.

1. Confirm the cause on the affected pod.

kubectl describe pod <pod-name> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

2. Check live memory against the limit.

kubectl top pod <pod-name> -n <namespace> --containers

DIAGNOSE. Look at the Last State block in the describe output for Terminated, OOMKilled, and Exit Code: 137. Compare resources.limits.memory to the working set from kubectl top and from your metrics over the last few hours or days. A flat line near the limit means the limit is undersized. A line that climbs and never plateaus means a leak, and raising the limit only delays the next kill.

FIX. If the limit is simply too low, set requests and limits deliberately. Setting them equal puts the pod in the Guaranteed QoS class and makes scheduling predictable.

resources:
  requests:
    memory: "512Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"
    cpu: "500m"

If the runtime ignores the cgroup limit, make it honor the container budget rather than host memory. For a modern JVM:

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:MaxRAMPercentage=75.0"

Container support is on by default in current JVMs. For Node.js, cap the old space below the container limit:

env:
  - name: NODE_OPTIONS
    value: "--max-old-space-size=384"

Keep the runtime cap comfortably under the container limit so native and off heap allocations still fit. If metrics show a true leak, raising the limit is a stopgap only. Profile the process, fix the leak, and then right-size.

How to prevent it

• Load test for memory, not just latency, and record peak working set.
• Set requests and limits on every container deliberately, never leave them empty.
• Monitor container working set memory and alert before it reaches the limit.
• Configure runtimes to read the cgroup limit (JVM MaxRAMPercentage, Node old space cap).
• Use the Vertical Pod Autoscaler in recommendation mode for guidance on sizing.

When to call a senior engineer

Call for help when usage climbs without plateau and you cannot locate the leak, when a runtime keeps overshooting the limit despite the flags above, or when OOM kills cascade across a node and take healthy pods with them. Erzon engineers can pull a heap profile, trace the allocation path, and right-size the deployment so it stays stable under real load.