Fix: DNS SERVFAIL

A dig query that comes back with status: SERVFAIL means the recursive resolver tried to answer and could not. Unlike NXDOMAIN, which is a definite answer that the name does not exist, SERVFAIL is the resolver telling you the resolution process itself broke. The failure can live in three different places: the authoritative nameservers for the zone, the DNSSEC trust chain, or the resolver you are asking. The fastest path to a fix is figuring out which of the three it is, and dig can do that in two or three queries.

What this error means

When you run a lookup you will see something like this:

dig example.com A

;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 31337
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

SERVFAIL (RCODE 2) is a catch-all from the resolver: it contacted, or tried to contact, the authoritative servers for the zone and could not produce an answer it trusts. The three big buckets are authoritative servers that are down or misconfigured, a DNSSEC chain that fails validation, and a resolver that is itself broken or unreachable upstream. Because the error is generated by the resolver rather than the zone, the same domain can SERVFAIL on one network and resolve fine on another.

Common causes

• All authoritative nameservers for the zone are down, unreachable, or refusing queries.
• Lame delegation: the parent zone points at nameservers that do not actually serve the zone.
• DNSSEC validation failure: expired RRSIG signatures, a DS record at the parent that does not match the zone’s keys, or a zone that dropped DNSSEC without removing its DS record.
• Resolver misconfiguration: a forwarder pointing at a dead upstream, a broken conditional forward zone, or a firewall blocking outbound port 53 or large UDP responses.
• An outage at the upstream resolver your servers use, so every lookup fails regardless of domain.

How to fix it

STABILIZE first. If production hosts cannot resolve anything, point them at a known-good public resolver so services come back while you investigate.

# Temporary: swap the resolver on an affected host
echo "nameserver 1.1.1.1" | sudo tee /etc/resolv.conf

If only your own domain is failing for the world, and your zone is signed, the fastest stabilization for a DNSSEC failure is to resign the zone or pull the DS record at the registrar. Details below.

DIAGNOSE. First, separate a DNSSEC failure from everything else. The +cd flag (checking disabled) tells the resolver to skip validation:

dig example.com A @1.1.1.1
dig example.com A @1.1.1.1 +cd

If the plain query returns SERVFAIL and the +cd query returns an answer, this is a DNSSEC validation failure, full stop. Confirm the broken link in the chain with delv, which validates locally and prints where the chain fails:

delv example.com A

Look for output like validating example.com/A: no valid signature found or an expired RRSIG. You can inspect signature dates directly:

dig example.com A +dnssec @1.1.1.1 +cd

In the RRSIG record, the two timestamps are expiration and inception. If the expiration is in the past, the zone was not resigned in time.

If +cd also fails, DNSSEC is not the problem. Check whether the authoritative servers answer at all. Walk the delegation:

dig example.com A +trace

Then query each authoritative server directly:

dig NS example.com @1.1.1.1 +cd
dig example.com A @ns1.example.com +norecurse

Timeouts or REFUSED from every listed nameserver means the authoritative side is down or the delegation is lame. If the authoritative servers answer fine when queried directly, but your resolver still fails while public resolvers succeed, the problem is your resolver: check its forwarders, its logs, and whether the network path to its upstream is open.

# Compare resolvers to localize the fault
dig example.com A @8.8.8.8
dig example.com A @1.1.1.1
dig example.com A          # uses /etc/resolv.conf

FIX. Match the fix to the bucket you found:

• Expired signatures: resign the zone. Managed DNS providers resign automatically, so an expiry usually means a self-hosted signer (BIND, Knot, PowerDNS) whose resigning job broke. Fix the job, resign now, and confirm with delv.
• DS mismatch or DNSSEC abandoned: either publish DNSKEY records that match the DS at the parent, or remove the DS record at the registrar. After removing the DS, resolution recovers as the DS TTL expires from caches.
• Dead authoritative servers: restore the servers, or if they are gone for good, update the NS records at the registrar to nameservers that actually host the zone.
• Broken local resolver: fix the forwarder address, open outbound UDP and TCP port 53 in the firewall, and make sure EDNS responses are not being dropped by an old middlebox. Restarting the resolver clears poisoned or stale state.
• Upstream resolver outage: switch /etc/resolv.conf, DHCP options, or your VPC DNS settings to an alternate resolver, and add a second nameserver line so a single upstream outage does not take you down again.

Verify the fix from multiple vantage points, not just your own machine, because caches recover at different times:

dig example.com A @8.8.8.8 +short
dig example.com A @1.1.1.1 +short
dig example.com A @9.9.9.9 +short

How to prevent it

• Run at least two authoritative nameservers on separate networks or providers, so one failure cannot make the zone unreachable.
• If you sign your zone, monitor RRSIG expiration and alert at least a week before the earliest signature expires.
• Treat DS record changes at the registrar as production changes: plan them, and never rotate keys without following the RFC 6781 rollover timing.
• Configure two upstream resolvers on every host and in every VPC, not one.
• Use an external DNS monitor (or a simple cron running dig against public resolvers) so you learn about SERVFAIL before your users do.

When to call a senior engineer

Call for help when a DNSSEC key rollover has gone wrong and you are not sure which of DS, DNSKEY, or RRSIG is the broken link, when resolution fails intermittently and only for some networks, or when the domain is down and every minute of DNS TTL math matters. Erzon engineers can walk the delegation, isolate the exact failing record, and get resolution restored while explaining what to change so it does not happen again.