Incident Management

Why Your PodDisruptionBudget Didn’t Save the Last EKS Upgrade

by
Why Your PodDisruptionBudget Didn’t Save the Last EKS Upgrade

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering A PodDisruptionBudget doesn’t protect your application — it protects a number. That’s why a routine EKS managed node group upgrade recently stalled on us for 40 minutes and paged on-call at 2am, even though our minAvailable: 2 config was textbook. The PDB was doing exactly what … Read more

During a Production Failure, the Real Issue Is Often Not Where the Error Is Showing

by
During a Production Failure, the Real Issue Is Often Not Where the Error Is Showing

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering Here is something nobody tells you when you start in DevOps: the error message is almost never the problem. It is just the messenger. A pod crashes — you blame the application. An API times out — you blame the network. A deployment fails — you … Read more

We Lost 3 Hours of Production Deployments Because of One Silent Node Provisioning Failure

by
We Lost 3 Hours of Production Deployments Because of One Silent Node Provisioning Failure

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering We were in the middle of a production scaling event when everything went quiet — in the worst way possible. No crash. No alert. No obvious Kubernetes error. Just pods stuck in Pending, GitHub Actions deployment jobs timing out, and the entire team staring at dashboards … Read more