Why Your PodDisruptionBudget Didn't Save the Last EKS Upgrade - TZoneLabs Why Your PodDisruptionBudget Didn't Save the Last EKS Upgrade

By KP | TZoneLabs | DevOps & Cloud Engineering

A PodDisruptionBudget doesn’t protect your application — it protects a number. That’s why a routine EKS managed node group upgrade recently stalled on us for 40 minutes and paged on-call at 2am, even though our minAvailable: 2 config was textbook. The PDB was doing exactly what it was told to do. That was the problem.

This post walks through why the math broke, how EKS node group upgrades actually work under the hood, and what we changed so a “correct” PDB stops silently blocking drains.

What a PDB Actually Guarantees

A PodDisruptionBudget doesn’t protect your application. It protects a number.

minAvailable: 2 tells the Kubernetes eviction API one thing: never let a voluntary eviction bring the number of ready pods for this selector below 2. That’s it. It has no idea which nodes those pods sit on, how many nodes are being drained at once, or whether the capacity plan makes sense.

When kubectl drain — or the EKS managed node group upgrade controller, which uses the same eviction API under the hood — tries to evict a pod, it doesn’t just delete it. It sends a request to the Eviction API, which checks the pod’s PDB before approving. If evicting that pod would violate minAvailable, the API returns a 429 and the eviction is denied. The draining process retries on a loop until either the eviction succeeds or a timeout is hit.

That retry loop is the stall. From the outside it looks like the upgrade is frozen. From the API’s perspective, it’s behaving exactly as configured.

Why the Math Broke

Here’s the setup that caused the incident: three replicas of a deployment, spread across two nodes because there was no topology spread constraint forcing better distribution. Two replicas landed on Node A, one on Node B.

Node A comes up for the managed node group upgrade. The drain controller tries to evict both pods on it. Evicting either one individually would drop total availability to 2, which the PDB allows. But evicting both at once — which is what draining a node requires — would drop it to 1. The PDB blocks it.

The node can’t fully drain until the two pods on it are gone. But it can’t evict them without violating the budget. So it waits, and keeps waiting, and keeps waiting.

⚠️ This isn’t a misconfigured number.
minAvailable: 2 is a perfectly reasonable value for a 3-replica deployment. The actual problem is that pod placement wasn’t PDB-aware. Two-thirds of the deployment’s capacity was sitting on one node, and PDB math assumes replicas are already spread in a way that makes eviction possible.

How EKS Node Group Upgrades Actually Work

It helps to know the mechanics here instead of treating it as a black box.

When you trigger a managed node group upgrade (new AMI, new launch template version, or a Kubernetes version bump), EKS does roughly this per node:

Cordon the node so no new pods schedule onto it.
Call the eviction API for every pod on the node, respecting PDBs.
Wait for pods to reschedule elsewhere and become ready.
Once the node is empty, terminate it and bring up a replacement.

Step 2 is where PDBs bite. If eviction is denied, the drain retries for a bounded period — the node group’s configured timeout, often 15 to 60 minutes depending on how it’s set — before EKS gives up on graceful drain and force-terminates the node anyway, PDB or not. That’s the “stuck for 40 minutes, then finally moves” pattern. It’s not stuck forever, but it’s stuck long enough to break a maintenance window and page someone.

What Actually Prevents This

None of these are exotic. They’re just easy to skip until something stalls at 2am.

1. Spread replicas with topology spread constraints, not just replica count

A PDB assumes reasonable distribution across nodes. We enforce that assumption instead of hoping for it:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: my-service

With maxSkew: 1 across a node-level topology key, the scheduler won’t let one node hold two replicas while another holds none — exactly the situation that stalled the drain above.

2. Size the PDB relative to actual replica count, not a copy-pasted default

A PDB of minAvailable: 2 on a 3-replica deployment leaves exactly one pod of slack. That’s tight enough that a single unlucky placement can block a drain. If the deployment can tolerate it, maxUnavailable: 1 on higher replica counts gives the drain controller more room to work with.

3. Test upgrades against a canary node group first

We roll the new AMI or version to a small, isolated node group before touching the fleet, and watch actual drain time. If a drain takes 40 minutes on a 3-node canary, it’ll take proportionally longer — and page proportionally louder — across the full node group.

4. Alert on drain duration, not just pod health

Pod health checks won’t catch this because, individually, every pod involved is healthy. The failure signal is a node stuck in NotReady,SchedulingDisabled for longer than expected, or an EKS node group update stuck in UPDATING past a threshold. Wire an alert to that condition specifically so someone gets paged while there’s still time to intervene manually, instead of waiting for the force-terminate to eventually kick in.

Key Lessons

A PDB is a constraint, not a guarantee.
It tells Kubernetes what it’s not allowed to do to your pods. It says nothing about whether replica count, pod placement, and node group size are consistent with each other.
PDB math assumes even distribution — it doesn’t enforce it.
Without a topology spread constraint, replicas can cluster on one node, making the budget mathematically unsatisfiable during a drain.
A “stuck” upgrade is often just a bounded retry loop.
Node drains have a timeout before EKS force-terminates. It resolves eventually — just late enough to break a maintenance window.
Canary the node group, not just the workload.
Drain time is a property of placement plus PDB config. The only way to see it is to actually drain a small node group and time it.
Changing the PDB number doesn’t fix this.
Making sure the assumptions a PDB depends on — even distribution, sane replica-to-node ratios — hold true is what fixes it.

Summary

Cause	What Happened	Fix
Uneven pod placement	2 of 3 replicas landed on one node	`topologySpreadConstraints` with `maxSkew: 1`
Tight PDB slack	`minAvailable: 2` on 3 replicas left no room to evict 2 at once	Size PDB relative to node-level replica clustering, not just total count
No drain-time visibility	Stall looked like a frozen upgrade, not a retry loop	Alert on `NotReady,SchedulingDisabled` duration
Untested upgrade path	First real drain timing seen was in production	Canary node group before fleet-wide rollout

The PDB never failed. It enforced its rule correctly the entire time — the rule just wasn’t compatible with how the pods happened to be placed.

Why Your PodDisruptionBudget Didn’t Save the Last EKS Upgrade