Setting Up Kubernetes Cluster Autoscaler on EKS: The Settings That Actually Matter

Setting Up Kubernetes Cluster Autoscaler on EKS: The Settings That Actually Matter

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering Installing Cluster Autoscaler is one Helm command. Getting it to scale up fast enough and scale down without evicting something it shouldn’t is the part that takes actual configuration. The defaults are conservative on purpose, and left untouched they either leave pending pods waiting longer than … Read more

Our Rolling Deploys Were Sending Live Traffic to Pods That Weren’t Ready, Because the Readiness Probe Checked the Wrong Port

Our Rolling Deploys Were Sending Live Traffic to Pods That Weren’t Ready, Because the Readiness Probe Checked the Wrong Port

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering Every rolling deploy for two months sent a burst of 502 errors to a small percentage of users, for about 30 seconds each time. Support tickets blamed “the app being slow sometimes.” The actual cause: our readiness probe was checking a port the container wasn’t listening … Read more

Our TLS Certificate Expired in Production Because Every Alert We Had Was Watching the Wrong Thing

Our TLS Certificate Expired in Production Because Every Alert We Had Was Watching the Wrong Thing

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering At 3 AM, our ingress started rejecting every HTTPS request with a certificate error. Every health check we had was green. Every pod was Running. The cluster looked perfectly healthy because none of our monitoring ever asked the one question that mattered: how many days is … Read more

Kubernetes RBAC Setup: A Practical Guide (No More Guessing Permissions)

Kubernetes RBAC Setup: A Practical Guide (No More Guessing Permissions)

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering The fastest way to grant a pod too much power in Kubernetes is kubectl create clusterrolebinding temp –clusterrole=cluster-admin –serviceaccount=default:default. It works immediately, which is exactly the problem. Nobody circles back to fix it, and six months later a compromised pod can read every secret in the … Read more

Why Your PodDisruptionBudget Didn’t Save the Last EKS Upgrade

Why Your PodDisruptionBudget Didn’t Save the Last EKS Upgrade

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering A PodDisruptionBudget doesn’t protect your application — it protects a number. That’s why a routine EKS managed node group upgrade recently stalled on us for 40 minutes and paged on-call at 2am, even though our minAvailable: 2 config was textbook. The PDB was doing exactly what … Read more

Your EKS Nodes Run Out of IPs, Not CPU: A Practical Guide to VPC CNI Warming and Prefix Delegation

Your EKS Nodes Run Out of IPs, Not CPU: A Practical Guide to VPC CNI Warming and Prefix Delegation

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering Most engineers size EKS nodes by CPU and memory. We did too — until one day a pod sat in Pending, the node had plenty of compute free, and the events said something about failing to assign an IP. That was our introduction to the quietest … Read more

During a Production Failure, the Real Issue Is Often Not Where the Error Is Showing

During a Production Failure, the Real Issue Is Often Not Where the Error Is Showing

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering Here is something nobody tells you when you start in DevOps: the error message is almost never the problem. It is just the messenger. A pod crashes — you blame the application. An API times out — you blame the network. A deployment fails — you … Read more

We Lost 3 Hours of Production Deployments Because of One Silent Node Provisioning Failure

We Lost 3 Hours of Production Deployments Because of One Silent Node Provisioning Failure

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering We were in the middle of a production scaling event when everything went quiet — in the worst way possible. No crash. No alert. No obvious Kubernetes error. Just pods stuck in Pending, GitHub Actions deployment jobs timing out, and the entire team staring at dashboards … Read more