Fix: x509: certificate signed by unknown authority

If a client is failing with x509: certificate signed by unknown authority, it completed the TLS handshake far enough to see the server’s certificate but could not link that certificate to a certificate authority it trusts, so it refused the connection. This is a trust problem, not proof that the certificate is fake or expired. It shows up most in Go services and in containers because both verify strictly and often ship without a complete set of trusted roots. The right fix depends on whether the server is sending an incomplete chain or the client is missing a root, and you should never resolve it by turning verification off.

Get "https://internal-api.example.com/v1/health": x509: certificate signed by unknown authority

What this error means

A server certificate is signed by an intermediate CA, which is signed by a root CA that the client already trusts. To verify the server, the client builds a chain from the leaf certificate up to a trusted root. This error means the client could not complete that chain: either the server did not present the intermediate needed to reach a known root, or the client’s trust store does not contain the root (or internal CA) at the top of the chain. Go’s TLS stack in particular does not fetch missing intermediates or cache them the way browsers do, so an incomplete chain that “works everywhere else” fails a Go client outright.

The certificate itself may be entirely valid. What is missing is a trusted path to it.

Common causes

• A slim or scratch container image has no ca-certificates installed, so the client trusts no public roots at all.
• The server presents only its leaf certificate and omits the intermediate(s), leaving an incomplete chain.
• An internal or corporate CA signed the certificate and that CA is not installed in the client’s trust store.
• A self-signed certificate is in use and has not been distributed to clients.
• The runtime is pointed at the wrong bundle, so the OS trust store is fine but the app cannot see it.
• A TLS-inspecting proxy re-signs traffic with a root the client does not trust.

How to fix it

STABILIZE first, without weakening security. If a job or deploy is blocked, install the correct trust material rather than disabling verification. In most containers, adding the CA bundle is enough to unblock:

apk add --no-cache ca-certificates    # alpine
# or, on debian slim
apt-get update && apt-get install -y ca-certificates

Do not reach for InsecureSkipVerify, curl -k, or NODE_TLS_REJECT_UNAUTHORIZED=0. Those disable the check that proves you are talking to the real server and turn a trust bug into a security hole. Use them only as a momentary diagnostic, never as the fix.

DIAGNOSE which side is at fault by reading the chain the server actually presents:

openssl s_client -connect internal-api.example.com:443 -servername internal-api.example.com -showcerts

Read the certificate list and the Verify return code at the bottom. If the server sends only one certificate (the leaf) with no intermediate, or you see code 21 (unable to verify the first certificate), the server chain is incomplete: fix the server. If the chain is complete but this client still fails, the client is missing the root or internal CA: fix the client. Confirm where your client looks for its bundle so you patch the right layer:

openssl version -d
ls -l /etc/ssl/certs/ca-certificates.crt

FIX by cause.

1. Missing client bundle (container or minimal host). Install and refresh the trust store, or copy a bundle into a scratch image at build time:

apt-get update && apt-get install -y ca-certificates && update-ca-certificates
# scratch image: copy the host bundle into the build
# COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

2. Internal or corporate CA not trusted. Add the CA to the system trust store so all clients on the host trust it:

sudo cp internal-root-ca.crt /usr/local/share/ca-certificates/internal-root-ca.crt
sudo update-ca-certificates

For a single process without touching the system store, point the client at a bundle that includes the CA:

export SSL_CERT_FILE=/path/to/bundle-with-internal-ca.pem

3. Incomplete server chain (server-side fix). Rebuild the served certificate so it includes the leaf followed by the intermediate(s), then reload. For nginx, ssl_certificate must point at the full chain, not the leaf alone:

cat leaf.crt intermediate.crt > fullchain.pem
sudo nginx -t && sudo systemctl reload nginx

4. Verify the fix from the client side with the trust store it will actually use:

curl -v https://internal-api.example.com/v1/health

How to prevent it

• Always serve the full chain (fullchain.pem), never the leaf certificate alone, so strict clients like Go succeed.
• Install and refresh ca-certificates in every image, including alpine, distroless, and scratch builds.
• Distribute internal and corporate CAs through the system trust store, never by disabling verification.
• Pin SSL_CERT_FILE or the runtime bundle path in configuration so processes point at the right store deliberately.
• Add an automated check that runs openssl s_client against your endpoints so an incomplete chain is caught before clients are.
• Never ship InsecureSkipVerify or equivalent flags to production, and lint for them in review.

When to call a senior engineer

Call for help when the chain looks correct in openssl yet Go or container clients still reject it, when an internal PKI or TLS-inspecting proxy makes the trust picture ambiguous across many services, or when the pressure to disable verification is mounting under a deadline. Erzon engineers can trace exactly which layer, from server chain to container image to runtime bundle, is breaking trust, and fix it without weakening TLS, so your services verify cleanly and stay secure.