Our Rolling Deploys Were Sending Live Traffic to Pods That Weren't Ready, Because the Readiness Probe Checked the Wrong Port - TZoneLabs Our Rolling Deploys Were Sending Live Traffic to Pods That Weren't Ready, Because the Readiness Probe Checked the Wrong Port

By KP | TZoneLabs | DevOps & Cloud Engineering

Every rolling deploy for two months sent a burst of 502 errors to a small percentage of users, for about 30 seconds each time. Support tickets blamed “the app being slow sometimes.” The actual cause: our readiness probe was checking a port the container wasn’t listening on yet, so Kubernetes marked every new pod Ready as soon as the probe timed out into a false pass, not when the app was actually able to serve traffic.

This post covers how a probe that looked correct in the YAML was quietly wrong at runtime, and how we caught it.

What Happened

The application had recently moved from listening on port 8080 to port 8081, as part of a change to run behind a sidecar proxy. The deployment’s containerPort and the Service’s targetPort were both updated correctly. The readiness probe was not.

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080  # stale — app now listens on 8081
  initialDelaySeconds: 5
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 3

With failureThreshold: 3 and a 2-second timeout, the probe would fail three times, then Kubernetes treated the pod as ready anyway once the pod’s total probe attempts exceeded a threshold tied to the deployment’s rollout configuration, which is not how readiness probes are supposed to behave, and was in fact a separate misconfiguration layered on top: someone had also set minReadySeconds low enough that the rollout controller moved on before confirming steady health.

Where We Started Looking

The issue never showed up as a full outage, just a short spike of errors correlated with deploys, which made it easy to dismiss as normal deploy noise.

Step 1 — Correlate error spikes with deploy events

kubectl get events --field-selector reason=ScalingReplicaSet \
  --sort-by='.lastTimestamp' -n production

Every 502 spike in our load balancer logs lined up, within a minute, with a ScalingReplicaSet event from a rolling deploy.

Step 2 — Check pod readiness timing during a deploy

kubectl get pods -n production -w

NAME                         READY   STATUS    RESTARTS   AGE
app-7d9f8c6b4-x2k9p          0/1     Running   0          3s
app-7d9f8c6b4-x2k9p          1/1     Running   0          6s

The pod went Ready six seconds after starting, but the application’s own startup logs showed it wasn’t actually accepting connections until roughly 12 seconds in.

Step 3 — Check the readiness probe’s actual target

kubectl describe pod app-7d9f8c6b4-x2k9p -n production | grep -A3 Readiness

Readiness:  http-get http://:8080/healthz delay=5s timeout=2s period=5s #success=1 #failure=3

Port 8080. The application had been listening on 8081 for two months. Every readiness check had been failing, silently, the entire time.

Step 4 — Confirm the probe was failing, not passing

kubectl logs app-7d9f8c6b4-x2k9p -n production --previous | grep -i "connection refused"

Nothing in the app logs, because a connection refused on a probe doesn’t show up in application logs at all, only in kubectl describe pod event history, which most dashboards don’t surface by default.

Why It Still Went Ready

🔴 Root cause: two settings compounding a wrong port.

The readiness probe pointed at the old port (8080) after the application moved to 8081.

minReadySeconds on the deployment was set to 0, so the rollout controller advanced to the next pod as soon as one probe cycle completed, pass or fail, instead of requiring a sustained pass.

The Service’s targetPort correctly pointed at 8081, so once a pod was marked Ready (correctly or not), real traffic was routed to it immediately.

The probe wasn’t silently passing. The rollout was silently not waiting for it to.

The Fix

Correct the probe port

readinessProbe:
  httpGet:
    path: /healthz
    port: 8081
  initialDelaySeconds: 8
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 3

Set minReadySeconds so a pod has to hold healthy, not just report healthy once

spec:
  minReadySeconds: 10
  strategy:
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1

maxUnavailable: 0 means the old pod doesn’t get removed until the new one has genuinely proven itself, not just been created.

What We Put in Place After

1. A CI check that diffs container ports against probe ports

CONTAINER_PORT=$(yq '.spec.template.spec.containers[0].ports[0].containerPort' deployment.yaml)
PROBE_PORT=$(yq '.spec.template.spec.containers[0].readinessProbe.httpGet.port' deployment.yaml)

if [ "$CONTAINER_PORT" != "$PROBE_PORT" ]; then
  echo "ERROR: readiness probe port ($PROBE_PORT) does not match container port ($CONTAINER_PORT)"
  exit 1
fi

2. An alert on deploy-correlated error spikes, not just sustained error rates

A 30-second spike during a deploy never crossed our existing error-rate alert threshold, which was tuned for sustained problems. We added a separate alert scoped to the deploy window specifically.

3. A synthetic check that hits the actual app port, not just the probe endpoint

The probe endpoint and the port it’s supposed to run on can drift independently. A synthetic monitor calling the real service from outside the cluster catches what an internal probe missed.

Key Lessons

A readiness probe pointed at the wrong port fails silently, not loudly.
It doesn’t crash the pod or show up in application logs. It just never reports true.
minReadySeconds: 0 means “ready” and “proven ready” are treated as the same thing.
They aren’t. A short sustained-health window catches problems a single probe cycle won’t.
Port changes during infrastructure migrations need a checklist, not memory.
The container port, Service target port, and probe port all have to move together, and nothing enforces that by default.
Short, deploy-correlated error spikes hide from sustained-error alerting.
If your alert threshold is tuned for ongoing problems, a 30-second spike every deploy will never trip it.
Test from outside the cluster, not just inside it.
An internal probe and an external synthetic check can disagree, and the disagreement is the signal.

Summary

Layer	What Happened	Tool to Check
Readiness Probe	Checked a stale port the app no longer listened on	`kubectl describe pod`
Rollout Strategy	`minReadySeconds: 0` let unhealthy pods receive traffic	Deployment spec
Service	Correctly routed to the new port, exposing the gap immediately	Service targetPort config
Load Balancer	502s during every rolling deploy, for two months	Load balancer access logs

Every layer here was individually configured on purpose. None of them were checked against each other.

Our Rolling Deploys Were Sending Live Traffic to Pods That Weren’t Ready, Because the Readiness Probe Checked the Wrong Port