DNS Troubleshooting for DevOps: dig, nslookup, and Common Failures

DNS Failures Are Not Always DNS Failures

"It's always DNS" is the meme. The frustrating reality is that DNS failures present as application errors, timeout exceptions, or "service unavailable" messages that give no indication of the actual cause. A developer will spend an hour debugging their application before someone runs a single dig command.

This guide is a practical reference for diagnosing DNS issues quickly in production — from basic resolution failures to split-horizon problems, negative caching, and Kubernetes CoreDNS weirdness.

The Essential Toolkit

# Install on Debian/Ubuntu
apt install dnsutils bind9-utils

# Install on RHEL/CentOS
yum install bind-utils

# Install on Alpine (containers)
apk add bind-tools

The four commands you'll use: dig, nslookup, resolvectl (systemd-resolved systems), and host.

dig: Your Primary Diagnostic Tool

nslookup is convenient but dig gives you the full picture. Learn the flags:

# Basic A record lookup
dig api.example.com

# Query a specific DNS server (bypass local resolver)
dig @8.8.8.8 api.example.com

# Short answer only
dig +short api.example.com

# Full trace from root servers (diagnose delegation issues)
dig +trace api.example.com

# Check TTL and caching behavior
dig +nocmd +noall +answer api.example.com

# Query specific record types
dig api.example.com MX
dig api.example.com TXT
dig api.example.com CNAME
dig api.example.com NS
dig api.example.com SOA

# Reverse DNS lookup (PTR record)
dig -x 93.184.216.34

# Check DNSSEC validation
dig +dnssec api.example.com

# Query over TCP (when UDP responses are truncated)
dig +tcp api.example.com

Understanding dig output:

; <<>> DiG 9.18.1 <<>> api.example.com
;; QUESTION SECTION:
;api.example.com.               IN      A

;; ANSWER SECTION:
api.example.com.        300     IN      A       93.184.216.34
                         ^^^
                         TTL in seconds — 300 = 5 minutes until cache expires

;; Query time: 12 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)   <- which server answered
;; WHEN: Sat Mar 29 2026
;; MSG SIZE  rcvd: 56

The AUTHORITY SECTION appears when you're getting a referral. The ADDITIONAL SECTION includes glue records. If you see NXDOMAIN in the status, the domain doesn't exist. If you see SERVFAIL, the authoritative nameserver has a problem.

Common Failure Modes and How to Diagnose Them

Failure 1: NXDOMAIN (Name Does Not Exist)

dig nonexistent.example.com
# Status: NXDOMAIN

Causes:

• Record genuinely doesn't exist
• Typo in the record name
• Record was deleted
• Negative caching is serving a cached NXDOMAIN

# Check if the record exists at the authoritative nameserver
# First find the authoritative NS
dig example.com NS +short

# Then query that specific server directly
dig @ns1.example.com nonexistent.example.com

# If the authoritative NS says NOERROR but your resolver says NXDOMAIN,
# you're seeing a cached negative response. Check the TTL on the SOA record's
# minimum field — that's how long NXDOMAIN is cached
dig example.com SOA +short
# Returns: ns1.example.com. admin.example.com. 2024010101 3600 900 604800 60
#                                                                          ^^
#                                                              60 seconds negative TTL

Failure 2: SERVFAIL (Server Failed)

dig api.example.com
# Status: SERVFAIL

Causes:

• Authoritative nameserver is unreachable
• DNSSEC validation failure
• Misconfigured zone delegation

# Bypass your resolver and query root/TLD/authoritative directly
dig +trace api.example.com

# Disable DNSSEC validation to check if that's the cause
dig +cd api.example.com   # +cd = checking disabled

# Query authoritative nameservers directly
dig api.example.com NS +short     # Get NS records
dig @<authoritative-ns> api.example.com    # Query them directly

If direct queries to the authoritative NS work but your resolver returns SERVFAIL, the problem is DNSSEC or resolver policy.

Failure 3: Wrong Answer (Stale Cache or Split-Horizon Mismatch)

The most insidious failure — DNS resolves, but to the wrong IP.

# Compare what different resolvers return
dig @8.8.8.8 api.example.com +short    # Google Public DNS
dig @1.1.1.1 api.example.com +short    # Cloudflare DNS
dig @<your-internal-dns> api.example.com +short  # Internal resolver

# Check if there's a split-horizon (different answer inside vs outside)
# From inside your network:
dig api.example.com +short
# From outside (use a web-based DNS checker or a VM with a public IP):
# Should return the same value or a clearly different one by design

Verify the current propagation state:

# Check multiple authoritative nameservers for consistency
for NS in $(dig example.com NS +short); do
  echo -n "${NS}: "
  dig @${NS} api.example.com +short
done

If nameservers return different answers, you have a zone transfer or replication lag issue.

Failure 4: DNS Resolution Works, But Slowly

# Time the resolution
time dig api.example.com

# Check if the answer is coming from cache (AA flag absent = cached response)
dig api.example.com
# Look for "aa" flag in the flags line — "aa" (authoritative answer) means
# you're hitting the authoritative server, no "aa" means cached

# High query time > 200ms from your resolver suggests:
# 1. Resolver is far from authoritative NS
# 2. Resolver has no cache warm-up
# 3. Network path issues

Failure 5: Kubernetes / CoreDNS Failures

Kubernetes DNS has its own failure modes. Pods can't resolve service.namespace.svc.cluster.local:

# Check CoreDNS pods are running
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Check CoreDNS logs for errors
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

# Verify CoreDNS ConfigMap
kubectl get configmap coredns -n kube-system -o yaml

# Test resolution from inside a pod
kubectl run dnstest --image=busybox:1.35 --rm -it --restart=Never -- \
  sh -c "nslookup kubernetes.default.svc.cluster.local"

# Test with dig for more detail
kubectl run dnstest --image=nicolaka/netshoot --rm -it --restart=Never -- \
  dig @10.96.0.10 kubernetes.default.svc.cluster.local

Common CoreDNS issues:

Symptom	Likely Cause	Fix
Only internal services fail	CoreDNS crash loop	Check OOM limits, increase memory
External DNS fails	Missing forward rule	Add forward . 8.8.8.8 to Corefile
5-second delay on first request	NDOTS=5 causing extra searches	Set ndots:1 in pod dnsConfig
Intermittent failures under load	CoreDNS CPU throttling	Remove CPU limits or increase them

The NDOTS problem specifically:

# Default Kubernetes pods have ndots:5, meaning for a name with < 5 dots,
# the resolver tries appending search domains first:
# api -> api.production.svc.cluster.local -> api.svc.cluster.local -> api.cluster.local -> api
# This causes 4 failed lookups before the real one for short names

# Override in your pod spec:
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "1"

Reference: dig Status Codes

Status	Meaning	First thing to check
NOERROR	Success	Is the answer correct?
NXDOMAIN	Domain doesn't exist	Typo? Record deleted? Negative cache?
SERVFAIL	Server failure	DNSSEC? Authoritative NS reachable?
REFUSED	Server refused query	Firewall? Resolver policy?
FORMERR	Format error in query	Dig version? Try +nocookie
NODATA	Domain exists but no record of that type	Wrong record type?

Quick Diagnostic Checklist

When something can't resolve:

# 1. Can you reach the DNS server at all?
nc -zv <DNS_SERVER_IP> 53

# 2. What resolver is the system using?
cat /etc/resolv.conf
resolvectl status   # systemd-resolved systems

# 3. Does the record exist at the authoritative NS?
dig +trace <hostname>

# 4. Is the problem specific to one record type?
dig <hostname> A
dig <hostname> AAAA
dig <hostname> CNAME

# 5. Is there a negative cache serving stale NXDOMAIN?
dig <hostname> | grep "ANSWER: 0"   # Zero answers = NXDOMAIN or NODATA

# 6. Compare internal vs external resolution
dig @8.8.8.8 <hostname> +short
dig @<internal-resolver> <hostname> +short

DNS is infrastructure that everything depends on silently. The best time to build your DNS troubleshooting muscle memory is before an outage, not during one. Run dig +trace on a few of your critical services right now and understand the delegation chain — that knowledge will save you hours when it matters.