We Rotated a Database Password, and Every Pod That Didn't Restart Started Failing Auth - TZoneLabs We Rotated a Database Password, and Every Pod That Didn't Restart Started Failing Auth

By KP | TZoneLabs | DevOps & Cloud Engineering

A scheduled quarterly rotation replaced the database credential in Secrets Manager, and within minutes most of the fleet was using it without issue. Two hours later, a subset of pods started failing every query with authentication errors — pods that hadn’t restarted in weeks, still holding the old password in an environment variable that was resolved once, at container start, and never looked at again.

This covers why env-var secrets don’t update on rotation, what mounted secret volumes actually sync (and what they don’t), and how to make a rotation runbook not depend on which pods happen to restart first.

What Happened

At 02:00 UTC, the security team rotated the shared database credential as part of a routine quarterly policy — nothing about the change itself was unusual, and it had been done before without incident. Deployments that had rolled out recently, and anything reading the secret from a mounted volume with an app that re-read the file, picked up the new value within the hour.

By 04:00 UTC, alerts started coming in for authentication failures against the database, but only from a subset of services. The confusing part was that the same service, running the same image, was failing on some pods and succeeding on others — which ruled out a bad deploy or a config error, and pointed at something pod-specific instead of code-specific.

Why Some Pods Never Noticed the Rotation

The pods failing were the ones that hadn’t restarted since before the rotation. Their credential was injected as an environment variable via secretKeyRef:

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: password

Kubernetes resolves valueFrom env vars exactly once, when the container starts. Updating the underlying Secret object doesn’t touch a running container’s environment at all — there’s no watch, no live sync, nothing. The only way a pod picks up a new value is to be recreated.

Secret volumes behave differently, but not as differently as it’s easy to assume. A mounted secret is synced to the pod’s filesystem periodically by the kubelet (governed by its configured sync frequency, roughly a minute by default) — but that only updates the file on disk. If the application read the file once at startup and cached the value in memory, exactly like the env var case, it’s still running on the old credential until it restarts.

The Fix: Make Rotation Force a Restart, Don’t Assume One

The actual fix wasn’t a Kubernetes feature — it was treating “restart every pod holding this secret” as a required step of the rotation, not an assumption:

kubectl rollout restart deployment -l uses-secret=db-credentials

For rotation triggered by an external system (Secrets Manager, Vault), Reloader watches Secret/ConfigMap objects and triggers a rolling restart on any Deployment annotated to depend on them:

metadata:
  annotations:
    reloader.stakater.com/auto: "true"

That closes the gap without anyone needing to remember a manual step during the next rotation.

What Made This Worse Than It Needed to Be

No overlap window. The old credential was invalidated the moment the new one was issued, instead of staying valid for a grace period. An overlap window turns “every pod must restart immediately” into “every pod restarts within its normal cycle,” which is a much smaller operational demand.
No alerting tied to the rotation event itself. The auth failures were caught by a generic error-rate alert, not anything that correlated with “we rotated a secret two hours ago” — that correlation was done by a person, after the fact.
Long-lived pods with no forced restart cadence. Anything that hadn’t redeployed in weeks was, by definition, running on whatever credential existed weeks ago.

Key Lessons

Env-var secrets are resolved once, at container start.
Rotating the underlying Secret object does nothing to a pod that’s already running.
Mounted secret volumes sync the file on disk, not the application’s memory.
An app that reads the file once at startup gets no benefit from the kubelet’s sync.
A rotation runbook needs an explicit restart step, or a tool that provides one.
Kubernetes will not do this for you.
Reloader (or an equivalent) turns “someone remembers to restart” into an automatic response to the Secret changing.
An overlap window where both the old and new credential are valid turns a hard deadline into a rolling one.
That’s the difference between every pod needing to restart in the next five minutes and every pod restarting on its normal cycle.

Summary

Decision	Do	Avoid
Secret delivery	Mount as a volume, or pair env vars with a forced-restart mechanism	Assume `secretKeyRef` env vars update live
Rotation runbook	Include an explicit rolling-restart step	Assume Kubernetes propagates the change
Automation	Use Reloader or equivalent annotation-triggered restarts	Rely on someone remembering to restart manually
Credential validity	Allow an overlap window for old + new	Invalidate the old credential immediately

We Rotated a Database Password, and Every Pod That Didn’t Restart Started Failing Auth