A CoreDNS Cache Setting Kept Routing gRPC Traffic to Pods That No Longer Existed After Every Deploy

By KP | TZoneLabs | DevOps & Cloud Engineering

Every deploy of one gRPC service left about 5% of calls failing for the next 30 to 40 seconds, then it cleared up on its own. The rollout itself looked clean, every new pod passed its readiness probe on schedule. The failing calls were connecting to pod IPs that had already been terminated, and nothing about the deploy process should have made that possible.

This covers how a CoreDNS cache setting kept serving dead pod IPs well after Kubernetes had already removed them, why it only showed up on one specific service, and what we changed so a routine deploy stops causing a predictable window of errors.

What Happened

The recommendation service is called over gRPC by several other services, using a headless Kubernetes Service so each client resolves the individual pod IPs directly and load-balances between them client-side, the standard pattern for gRPC since a normal ClusterIP doesn’t play well with long-lived HTTP/2 connections. Clients cache the resolved IP list and reuse connections until one fails, then re-resolve.

Every rolling deploy of this service produced the same short burst of Unavailable errors from calling clients, always in the 30 to 40 second range after the new pods came up, then nothing.

Where We Started Looking

Step 1 — Read what the client error actually said

rpc error: code = Unavailable desc = connection error: desc = "transport:
Error while dialing dial tcp 10.42.3.17:9090: connect: connection refused"

A specific IP, refusing connections outright. Not a timeout, not a DNS failure, an active refusal, which usually means something used to be listening there and now isn’t.

Step 2 — Confirm that IP doesn’t belong to anything currently running

kubectl get pods -n prod -o wide | grep recommendation-service

No pod in the current set had that address. Cross-referencing against the deploy’s pod history showed it belonged to a pod terminated as part of the same rollout, about 35 seconds earlier.

Step 3 — Query the headless service directly, more than once

# From inside a client pod, run back to back
dig recommendation-service.prod.svc.cluster.local +short
sleep 2
dig recommendation-service.prod.svc.cluster.local +short

The first query, run right after the old pod was gone, still returned its IP alongside the new ones. Kubernetes’ own Endpoints object had already dropped it. Something between the Endpoints object and the client was still handing out the old address.

Step 4 — Check the actual CoreDNS config

kubectl get configmap coredns -n kube-system -o yaml

.:53 {
    ...
    cache 30
    ...
}

A 30-second cache on the CoreDNS cache plugin, added at some point to cut down on the query volume from a chatty internal tool. The window our errors cleared up in, 30 to 40 seconds after each deploy, matched that number almost exactly.

Why the Rollout Itself Looked Clean

🔴 Root cause: CoreDNS caches answers on a timer, independent of how fast the underlying Endpoints actually change.

Kubernetes removed the terminated pod from the Endpoints object immediately, exactly as expected.

CoreDNS had already cached the previous answer, including that pod’s IP, and kept serving it for up to 30 seconds regardless of what the Endpoints object now said.

Client-side gRPC connections already open to that IP stayed open until a request failed, then the client re-resolved, got the same stale cached answer, and failed again, on a loop bounded by the cache’s remaining TTL, not by anything the deploy did.

Once the cache entry actually expired and a fresh query hit the live Endpoints object, the errors stopped immediately.

The readiness probes, the rollout strategy, and the Endpoints object all did exactly what they were supposed to. The staleness was happening one layer further down, in DNS.

The Fix

Immediate: lower the cache TTL

.:53 {
    ...
    cache 5
    ...
}

Five seconds instead of thirty. Still enough to absorb the original query-volume problem the cache was added for, short enough that a deploy’s staleness window becomes a non-issue in practice.

Real fix: give terminating pods a grace window instead of relying on DNS alone

lifecycle:
  preStop:
    exec:
      command: ["sleep", "15"]
terminationGracePeriodSeconds: 30

A preStop hook delays the actual shutdown signal to the container, so a pod that’s already been pulled from Endpoints keeps accepting and completing in-flight connections for a short window instead of refusing them outright. Combined with the shorter cache TTL, any client still holding a stale IP during that window gets served correctly instead of hitting a closed socket.

What We Put in Place After

1. An alert on error rate specifically during deploy windows

A generic error-rate alert was already in place but tuned for sustained problems, not a 30-second spike that resolves on its own. We added a check that correlates gRPC Unavailable spikes against recent deploy timestamps, so this exact pattern gets flagged even when it clears up before a human looks at it.

2. Documented the headless-service-plus-DNS-cache interaction for every gRPC service

Any service using client-side load balancing over a headless Service now gets checked against the cluster’s DNS cache TTL as part of its rollout runbook, instead of assuming Kubernetes’ own endpoint management is the only layer that matters.

3. Set a floor on how low the cache TTL can go before CoreDNS query load becomes a problem

We monitored CoreDNS CPU and query rate after the change to confirm five seconds didn’t reintroduce the load issue the original 30-second value was set to fix, rather than assuming lower is free.

Key Lessons

Kubernetes updating Endpoints immediately doesn’t mean DNS reflects it immediately.
A caching layer in between can hold a stale answer well past the point the cluster itself considers a pod gone.
Headless services with client-side load balancing inherit DNS caching behavior directly.
A ClusterIP hides this entirely since the virtual IP never changes; a headless service exposes pod IPs straight to whatever cache sits in the resolution path.
A cache setting tuned for one problem can quietly cause another.
The 30-second TTL was solving a real query-volume issue and nobody revisited it against services that deploy frequently.
“Connection refused” from a real IP is a strong signal to check what used to be there.
It usually means the target existed recently, not that the network path is broken.
A short preStop delay is cheap insurance against exactly this kind of staleness.
It doesn’t fix the caching layer, but it buys the time for stale references to catch up.

Summary

Layer	What Happened	Tool to Check
Kubernetes Endpoints	Updated immediately, exactly as expected	`kubectl get endpoints`
CoreDNS Cache	Served the old pod IP for up to 30 seconds after removal	`kubectl get configmap coredns`
gRPC Clients	Reused stale connections until failure, then re-resolved into the same stale cache	Client error logs, `dig` from inside a pod
Deploy Process	Behaved correctly throughout, wasn’t the actual cause	Rollout status, readiness probe logs

The deploy never did anything wrong. It just exposed a cache setting that had been fine right up until something needed it to keep up.