Networking

A Missing VPC Endpoint Routed Every Image Pull Through NAT Gateway, and Our AWS Bill Jumped 9x

by
A Missing VPC Endpoint Routed Every Image Pull Through NAT Gateway, and Our AWS Bill Jumped 9x

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering Our AWS bill for a new EKS environment came in nine times higher than the month before, and nothing about our traffic had changed. The entire difference was NAT Gateway data processing — a private-subnet VPC with no Gateway Endpoint for S3 was routing every ECR … Read more

A Namespace Relabel Silently Broke a NetworkPolicy Selector, and Cross-Namespace Traffic Started Timing Out

by
A Namespace Relabel Silently Broke a NetworkPolicy Selector, and Cross-Namespace Traffic Started Timing Out

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering A payments-team service started timing out on calls to a shared internal API, intermittently, with nothing in either side’s logs pointing at why. No connection refused, no TLS error, no 5xx. Just a hang until the client’s own timeout gave up. The pods were healthy, the … Read more

Your EKS Nodes Run Out of IPs, Not CPU: A Practical Guide to VPC CNI Warming and Prefix Delegation

by
Your EKS Nodes Run Out of IPs, Not CPU: A Practical Guide to VPC CNI Warming and Prefix Delegation

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering Most engineers size EKS nodes by CPU and memory. We did too — until one day a pod sat in Pending, the node had plenty of compute free, and the events said something about failing to assign an IP. That was our introduction to the quietest … Read more