Fix: could not connect to server: Connection refused (Cloud SQL)

An application that talks to Cloud SQL through the Cloud SQL Auth Proxy suddenly fails with could not connect to server: Connection refused on 127.0.0.1. Refused means exactly one thing: no process was listening on the host and port the client dialed. With Cloud SQL that puts the fault in one of a few places, the proxy is not running or listening elsewhere, the proxy started but cannot reach the instance, or the app is bypassing the proxy and the direct network path is closed. The error looks like a database problem, but the database is usually fine.

What this error means

The Cloud SQL Auth Proxy runs next to your application, listens on a local port such as 127.0.0.1:5432, and tunnels traffic to the instance over TLS after authorizing with IAM. When the app sees connection refused on localhost, the operating system rejected the TCP handshake because nothing was bound to that port. The database never saw the attempt.

If the app connects directly instead, over the instance’s public IP with authorized networks, or over private IP inside the VPC, refused means the packet arrived somewhere that was not listening: the instance is stopped, the wrong IP is configured, or a middlebox reset the connection. Note the distinction from a timeout. A firewall that silently drops packets produces a hang and then a timeout, while refused is an active rejection, which on GCP most often points at the local proxy or a stopped instance rather than at VPC firewall rules.

Common causes

• The Auth Proxy container or process is not running: it crashed, was never added to the pod, or a sidecar started after the app tried to connect.
• The proxy exited at startup because the Cloud SQL Admin API is disabled or its service account lacks the Cloud SQL Client role.
• The app dials a different port than the proxy listens on, common when one proxy serves multiple instances on separate ports.
• The instance is stopped, or its activation policy left it off after a manual stop.
• Direct public IP connections from an address that is not in authorized networks, or direct private IP from a network without VPC peering or a route to the instance.
• The instance is mid-restart from a crash, maintenance, or a flags change, so nothing is accepting connections for a short window.
• PostgreSQL hit max_connections. That returns a FATAL too many clients error rather than refused, but it often appears in the same incident when the app retries in a tight loop.

How to fix it

STABILIZE. Confirm the instance itself is up before touching anything else:

gcloud sql instances describe INSTANCE_NAME \
  --project=PROJECT_ID \
  --format="value(state, settings.activationPolicy, ipAddresses)"

State must be RUNNABLE and the activation policy ALWAYS. If it was stopped, start it:

gcloud sql instances patch INSTANCE_NAME --activation-policy=ALWAYS

If the instance is up and the app connects through the proxy, restart the proxy or the pod so a clean proxy comes up, and watch its logs while it starts. A healthy start logs that it is listening; a broken one tells you why.

DIAGNOSE. Check that something is actually listening where the app is dialing:

ss -ltnp | grep 5432

If nothing is listening, read the proxy logs. The three classic startup failures are the Admin API being disabled, missing IAM, and a wrong instance connection name. Verify each:

gcloud services list --enabled --project=PROJECT_ID | grep sqladmin

gcloud projects get-iam-policy PROJECT_ID \
  --flatten="bindings[].members" \
  --filter="bindings.members:PROXY_SA_EMAIL" \
  --format="table(bindings.role)"

gcloud sql instances describe INSTANCE_NAME --format="value(connectionName)"

The proxy’s identity needs roles/cloudsql.client. On GKE with Workload Identity, remember the identity is the linked Google service account, not the Kubernetes one. For direct connections instead of the proxy, check the path: for public IP, confirm your egress address is present in settings.ipConfiguration.authorizedNetworks from the describe output; for private IP, confirm the client is in a VPC that peers with the instance’s network and that the instance actually has a private address.

FIX. Repair whichever link was broken. Enable the API and grant the role if missing:

gcloud services enable sqladmin.googleapis.com --project=PROJECT_ID

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:PROXY_SA_EMAIL" \
  --role="roles/cloudsql.client"

Run the proxy (v2) with an explicit port so app config and proxy config cannot drift, and add the private IP flag when the instance should be reached inside the VPC:

./cloud-sql-proxy --port 5432 PROJECT_ID:REGION:INSTANCE_NAME

./cloud-sql-proxy --private-ip --port 5432 PROJECT_ID:REGION:INSTANCE_NAME

For direct public IP access, add the client network deliberately and keep the list tight:

gcloud sql instances patch INSTANCE_NAME \
  --authorized-networks=203.0.113.0/24

Note that patch replaces the whole authorized networks list, so include every CIDR you still need. If the incident also produced too many clients errors, fix the pool before raising limits: cap the application pool size across all replicas so the sum stays under the instance’s connection ceiling. Raising max_connections via database flags restarts the instance, so treat that as a scheduled change, not an incident response:

gcloud sql instances patch INSTANCE_NAME --database-flags=max_connections=200

How to prevent it

• Run the proxy as a proper sidecar with a startup order guarantee, so the app container cannot start dialing before the proxy listens.
• Add a readiness check on the proxy port so orchestration never routes traffic to a pod whose tunnel is down.
• Grant roles/cloudsql.client through infrastructure as code, so a recreated service account does not silently lose it.
• Prefer private IP with the proxy or the language connectors over public IP with authorized networks, which rot as office and NAT addresses change.
• Size connection pools deliberately: pods multiplied by pool size must stay below the instance connection limit with headroom for migrations and admin sessions.
• Alert on the Cloud SQL instance state and on proxy restarts, not just on application error rate.

When to call a senior engineer

Call for help when connections are refused intermittently and you cannot correlate it with restarts or maintenance, when a Workload Identity chain on GKE refuses to authorize the proxy despite the bindings looking right, or when the real problem is connection exhaustion and the fix means redesigning pooling across dozens of services. Erzon engineers can trace the full path from the app socket through the proxy, IAM, and VPC to the instance, and leave you with a connection layer that survives failovers and maintenance windows.