By KP | TZoneLabs | DevOps & Cloud Engineering
Most engineers size EKS nodes by CPU and memory. We did too — until one day a pod sat in Pending,
the node had plenty of compute free, and the events said something about failing to assign an IP. That was
our introduction to the quietest capacity limit in EKS: the number of IP addresses a node can hand out.
We want to walk through how this actually works, the settings you can turn to control it, and prefix
delegation — the one knob most tutorials skip, and the one that changed how we pack nodes in production.
Why a pod needs an IP in the first place
On EKS with the AWS VPC CNI, every pod gets a real IP address from your VPC subnet. Not an overlay
address, not something NAT’d behind the node. An actual routable IP out of the same CIDR your EC2 instances
live in. That design is great for networking simplicity and security group support, but it means your pod
count is capped by how many IPs the node can carry.
Those IPs come from elastic network interfaces (ENIs) attached to the instance. Each instance type
supports a maximum number of ENIs, and each ENI supports a maximum number of IPs. Multiply the two,
subtract a bit of overhead, and you have your ceiling.
The math on a t3.medium
Take a t3.medium, which is a common default and what we run in both node groups. It
supports 3 ENIs, and each ENI gives you 6 IPs, of which 5 are usable for pods (one per ENI is reserved for
the node itself).
So the max pods calculation is:
(ENIs × usable IPs per ENI) + 2
(3 × 5) + 2 = 17That plus-2 covers host networking pods like aws-node and kube-proxy that do
not consume a secondary IP. The practical takeaway: a t3.medium tops out around 17 pods. You
can have 60% of your CPU idle and still be unable to schedule pod number 18. The scheduler is not confused.
The node is genuinely full of IPs.
This is where people lose an afternoon. They see Pending pods, check CPU, check memory,
find both healthy, and start blaming the scheduler. The answer was in the ENI allocation the whole time —
the same pattern we chased for three real hours in
We Lost 3 Hours of Production Deployments Because of One Silent Node Provisioning Failure,
just one layer further down the stack.
How the CNI decides when to grab IPs
Assigning an IP to a pod is not free. If there is no warm IP sitting on the node when a pod lands, the
CNI has to make a live AWS API call (AssignPrivateIpAddresses) before the pod can start. That
call takes time, and the pod waits on it. Multiply that across a scale-up event and your startup latency
gets ugly.
To avoid that, the VPC CNI keeps a pool of IPs ready in advance. You control the size of that pool with
three environment variables on the aws-node daemonset.
WARM_ENI_TARGET keeps N whole spare ENIs attached and ready. It is the coarse option,
because it reserves IPs a full ENI at a time. The default is 1, which is why a fresh node already holds
more IPs than it is using.
WARM_IP_TARGET keeps N free individual IPs ready regardless of ENI boundaries. It is finer
grained, and when both this and WARM_ENI_TARGET are set, WARM_IP_TARGET wins.
MINIMUM_IP_TARGET is a floor. It tells the CNI to always keep at least N IPs allocated even
when the pod count is low. This one is useful when you know a baseline burst is coming and you do not want
the first wave of pods waiting on API calls.
The trade-off nobody plans for
Warming IPs makes pods start faster. It also eats your subnet. Every warm IP is an address that is
allocated but not doing anything yet, and your subnet has a fixed pool.
A /24 private subnet gives you roughly 251 usable addresses. That sounds like a lot until
you remember it is shared across every pod, every node’s primary IP, and every node Karpenter launches on
demand. Two subnets at 10.0.1.0/24 and 10.0.2.0/24 can drain faster than you
expect once you run aggressive warming and let Karpenter scale nodes freely. IP exhaustion in a subnet is a
real, painful outage, and it does not announce itself until you are already out — a good reminder that
the real issue in production is often not where the error is showing.
So the tuning problem is a balance. Warm too little and pods wait on API calls during scale-up. Warm too
much and you exhaust the subnet. On a tight /24, that balance is narrow.
Prefix delegation, the setting that changes the shape of the problem
Here is the part most tutorials skip, and the part that mattered most once we started packing nodes more
densely.
Instead of assigning IPs one at a time, the CNI can assign them in /28 prefixes, which is
16 IPs in a single block, to each ENI. You turn this on with ENABLE_PREFIX_DELEGATION = true.
Run the density math again with prefixes on that same t3.medium:
(3 ENIs × (5 usable slots × 16 IPs per prefix)) + 2 ≈ 242The theoretical ceiling jumps from 17 to well over 200 pods on the exact same instance. That number is
real, but do not chase it. In practice you cap maxPods somewhere near the standard Kubernetes
recommendation of about 110 per node, because kubelet, conntrack tables, and the CNI daemon all get
stressed long before you hit 242 pods on a small instance.
The real value of prefix delegation is not cramming hundreds of pods onto a small node. It is two other
things. First, you stop wasting IPs by warming in efficient 16-address blocks instead of dribbling them out
one at a time. Second, and this is the big one if you run Karpenter, you stop launching brand-new nodes
just because an existing node ran out of addresses while its CPU and memory sat idle. Prefix delegation
lets Karpenter bin-pack tightly instead of spraying new instances to solve what was only ever an IP
shortage.
WARM_PREFIX_TARGET controls how many spare prefixes to keep ready, the same idea as
WARM_IP_TARGET but in blocks of 16. MAX_ENI caps how many ENIs a node will attach
(default -1, meaning use the instance’s own maximum), which is worth setting so you do not
silently reserve prefixes you never needed.
What to know before you flip it on
Prefix delegation is not a free switch. Two things will bite you if you skip the homework.
It only works on Nitro-based instances. The t3, m5, c5 families
and anything newer are fine. Older instance families are not, and the CNI will simply ignore the setting
there.
It needs contiguous /28 blocks. AWS has to find 16 addresses in a row to hand out a prefix.
On a long-lived, fragmented subnet where addresses have been allocated and freed many times, AWS may fail
to find a clean block, and you get allocation failures that look a lot like the exhaustion problem you were
trying to solve. The fix is to plan your addressing up front and, where you can, provision fresh subnets
for prefix-delegated node groups rather than retrofitting a crowded one.
Where this lives in the code
If you manage EKS with Terraform, none of this needs a manual kubectl set env on the
daemonset. The VPC CNI is an EKS addon, and you pass these settings through its
configuration_values as JSON in the addon block.
A vpc-cni entry that only pins a version and ordering looks like this:
vpc-cni = {
before_compute = true
addon_version = "v1.18.x-eksbuild.x"
}To turn on prefix delegation and warm one spare prefix, extend it:
vpc-cni = {
before_compute = true
addon_version = "v1.18.x-eksbuild.x"
configuration_values = jsonencode({
env = {
ENABLE_PREFIX_DELEGATION = "true"
WARM_PREFIX_TARGET = "1"
}
})
}Keep before_compute = true so the CNI is configured before your nodes join, otherwise nodes
can come up with the old allocation behavior and you have to recycle them. If you’re also locking down
pod-to-pod traffic on these same clusters, see our walkthrough on
End-to-End Encryption on Amazon EKS with cert-manager
for the certificate side of the setup.
The lesson worth keeping
When Kubernetes tells us it cannot schedule a pod, we now check IP allocation before blaming CPU or the
scheduler. On the VPC CNI, “insufficient capacity” is very often “no free IPs on this node,” and the node’s
compute graphs will look perfectly healthy while it happens.
The warming knobs let you trade subnet space for faster pod starts. Prefix delegation changes the trade
itself, letting you use the IPs you already have far more efficiently and letting Karpenter pack nodes the
way you actually intended. On a small instance and a tight /24, that is often the difference
between adding nodes we did not need and running the fleet we already pay for.
Key Lessons
- A healthy CPU/memory graph does not mean a node has capacity.
IP allocation on the VPC CNI is a separate, silent ceiling — a node can be idle on compute and still refuse to schedule another pod. - Karpenter and the Cluster Autoscaler don’t know why a pod won’t schedule.
They see “insufficient capacity” and often launch a new node, when the real fix was freeing up IP allocation on the nodes already running. - Warming settings trade subnet space for pod start speed.
There is no free warm-up — only a balance we have to tune against the size of our CIDR. - Prefix delegation isn’t just a higher pod ceiling.
Its real value is letting Karpenter bin-pack nodes tightly instead of spraying new instances to solve what was only ever an IP shortage. - Contiguous
/28blocks aren’t guaranteed on a fragmented subnet.
Plan addressing before turning on prefix delegation, not after — retrofitting a crowded subnet invites the same allocation failures you were trying to fix.
Summary
| Setting | What It Does | Where We Set It |
|---|---|---|
WARM_ENI_TARGET |
Keeps N whole spare ENIs attached and ready | aws-node env / EKS addon config |
WARM_IP_TARGET |
Keeps N free individual IPs ready regardless of ENI boundaries | aws-node env / EKS addon config |
MINIMUM_IP_TARGET |
Floor of IPs always kept allocated, even at low pod count | aws-node env / EKS addon config |
ENABLE_PREFIX_DELEGATION |
Assigns IPs in 16-address /28 blocks instead of one at a time |
aws-node env / EKS addon config |
WARM_PREFIX_TARGET |
Keeps N spare prefixes (16-IP blocks) ready | aws-node env / EKS addon config |
MAX_ENI |
Caps how many ENIs a node will attach | aws-node env / EKS addon config |
The short version: IP exhaustion on the VPC CNI looks like a scheduling problem but is really a subnet
math problem, and prefix delegation is the setting that gives Karpenter room to pack nodes the way it was
always meant to.
Read Next
- Kubernetes Silently Evicted 60% of Our Production Pods — Here Is Why and How to Fix It
- We Lost 3 Hours of Production Deployments Because of One Silent Node Provisioning Failure
If posts like this are useful, the best way to catch new ones is to
follow along on LinkedIn —
that’s where new TZoneLabs guides get announced first.
Tags:
#Kubernetes #EKS #AWS #VPC #Networking #Karpenter #DevOps #Terraform