Fix: SNAT port exhaustion

SNAT port exhaustion is one of the classic silent Azure failures: outbound calls from App Service, Functions, or VMs behind a load balancer begin timing out or failing intermittently, while CPU, memory, and the remote service all look healthy. Each instance has a finite allocation of source network address translation ports for outbound connections to the internet, and an application that churns connections can exhaust its share within seconds of a traffic spike. New outbound flows then queue or die until ports are released.

SNAT port exhaustion

What this error means

Private compute in Azure reaches the internet through source NAT: the platform maps each outbound flow onto a public IP and port pair. A flow to a given destination IP and port needs its own SNAT port, the pool is shared and preallocated (an App Service instance gets 128 ports by default, load balancer outbound allocations depend on backend pool size), and ports linger in use for minutes after a connection closes because of idle timeouts. When an instance’s allocation is fully occupied, new outbound connections to the internet cannot be established. The failure surfaces in your app as timeouts, connection refused, or hung requests to external APIs and databases, with nothing wrong on the remote side.

Common causes

• Opening a new connection per request instead of reusing a pooled HttpClient or database connection pool, the number one cause by far.
• High outbound concurrency to a single destination IP and port, since SNAT ports are unique per destination tuple.
• Traffic growth or a retry storm multiplying outbound flow counts past the fixed per instance allocation.
• Relying on default outbound access or a load balancer without enough frontend IPs and tuned outbound rules.
• Long idle timeouts holding ports hostage for connections nobody is using.
• Scaling instance count down, which shrinks total available ports while traffic stays flat.

How to fix it

STABILIZE first. Scale out: adding instances immediately multiplies total SNAT allocation and spreads outbound flows. It is a stopgap that buys time, not the fix.

1. DIAGNOSE and confirm. On App Service, open Diagnose and solve problems and run the SNAT Port Exhaustion detector. On a Standard Load Balancer, pull the SNAT metrics directly:

az monitor metrics list \
  --resource /subscriptions/<sub-id>/resourceGroups/rg/providers/Microsoft.Network/loadBalancers/lb \
  --metric SnatConnectionCount \
  --interval PT1M \
  --filter "ConnectionState eq 'Failed'"

Failed SNAT connections above zero during your incident window is the confirmation.

2. FIX the connection churn, which is where the ports actually go. Reuse a single shared HttpClient (or the platform’s pooled client) instead of constructing one per call, enable keep alive, and pool database connections. In .NET, a per request new HttpClient() is the canonical cause; a static client or IHttpClientFactory is the cure. This one change routinely cuts SNAT usage by an order of magnitude.

3. FIX the outbound architecture for VNet integrated workloads by attaching a NAT Gateway, which allocates SNAT ports on demand from a much larger pool:

az network public-ip create -g rg -n natgw-ip --sku Standard
az network nat gateway create -g rg -n natgw \
  --public-ip-addresses natgw-ip \
  --idle-timeout 4
az network vnet subnet update -g rg \
  --vnet-name app-vnet --name app-subnet \
  --nat-gateway natgw

For App Service, enable VNet integration and route all outbound traffic through the integrated subnet so the NAT Gateway carries it.

4. For destinations inside Azure (SQL Database, Storage, Key Vault), take them off the SNAT path entirely with service endpoints or private endpoints, so those flows stop consuming internet SNAT ports at all.

5. Keep idle timeouts short (the NAT Gateway default of 4 minutes is sensible) so abandoned connections return their ports quickly.

How to prevent it

• Enforce connection reuse in code review: one pooled HTTP client per destination, pooled database connections, keep alive on.
• Route production outbound traffic through NAT Gateway rather than default outbound access, which Azure has been retiring for new deployments.
• Use private endpoints for Azure to Azure dependencies so internal traffic never touches SNAT.
• Alert on SNAT allocation and failed SNAT connections per instance, since exhaustion warns before it bites.
• Load test outbound heavy paths at production concurrency, watching SNAT metrics, before launch.

When to call a senior engineer

Call for help when failures stay intermittent after pooling fixes and nobody can tie them to a port count, when the outbound path mixes default outbound access, load balancer rules, and firewalls into something nobody fully maps, or when a NAT Gateway migration has to happen under live traffic. Erzon engineers can trace every outbound flow to its SNAT source, quantify exactly where the ports go, and rebuild the outbound architecture so connection spikes stop turning into mystery timeouts.