● DevOps · Kubernetes · AWS

Master Cloud Engineering
From Real-World Experience

Practical tutorials, deep dives, and battle-tested guides for engineers who build and ship in the cloud.

Browse by topic → AWS Kubernetes DevOps GitLab

Our Rolling Deploys Were Sending Live Traffic to Pods That Weren’t Ready, Because the Readiness Probe Checked the Wrong Port

Our Rolling Deploys Were Sending Live Traffic to Pods That Weren’t Ready, Because the Readiness Probe Checked the Wrong Port

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering Every rolling deploy for two months sent a burst of 502 errors to a small percentage of users, for about 30 seconds each time. Support tickets blamed “the app being slow sometimes.” The actual cause: our readiness probe was checking a port the container wasn’t listening … Read more

We Leaked a Production Database Password Into Build Logs for Three Weeks Because One CI Variable Wasn’t Marked Masked

We Leaked a Production Database Password Into Build Logs for Three Weeks Because One CI Variable Wasn’t Marked Masked

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering For three weeks, our staging database password was sitting in plain text in every CI job log a specific pipeline produced. Anyone with read access to the project, which included a few external contractors, could open a job log and copy it out. Nobody stole it, … Read more

How to Structure a GitLab CI Pipeline With Stages and Caching (So Jobs Stop Reinstalling Everything)

How to Structure a GitLab CI Pipeline With Stages and Caching (So Jobs Stop Reinstalling Everything)

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering The default .gitlab-ci.yml most teams start with runs npm ci or pip install in every single job, on every single commit. A pipeline with a build, test, and lint job doing the same install three times isn’t three times safer. It’s three times slower for no … Read more

We Burned Two Weeks of Cloud Budget Because Our Autoscaler Was Stuck in a Scale-Up, Scale-Down Loop

We Burned Two Weeks of Cloud Budget Because Our Autoscaler Was Stuck in a Scale-Up, Scale-Down Loop

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering For nine days, our cluster added and removed the same three worker nodes every six minutes. Every dashboard we watched stayed green the whole time. Pods were running, nodes were Ready, CPU never crossed 60%. Nobody noticed until the AWS bill landed about 40% higher than … Read more

Debugging AWS IAM Denials Without Guessing at Policy JSON

Debugging AWS IAM Denials Without Guessing at Policy JSON

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering An AccessDenied error from AWS tells you almost nothing. It names the action and sometimes the resource, but never which of the five places a permission could be blocked actually blocked it. We used to open the IAM console and read policies line by line, sometimes … Read more

Our TLS Certificate Expired in Production Because Every Alert We Had Was Watching the Wrong Thing

Our TLS Certificate Expired in Production Because Every Alert We Had Was Watching the Wrong Thing

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering At 3 AM, our ingress started rejecting every HTTPS request with a certificate error. Every health check we had was green. Every pod was Running. The cluster looked perfectly healthy because none of our monitoring ever asked the one question that mattered: how many days is … Read more

Kubernetes RBAC Setup: A Practical Guide (No More Guessing Permissions)

Kubernetes RBAC Setup: A Practical Guide (No More Guessing Permissions)

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering The fastest way to grant a pod too much power in Kubernetes is kubectl create clusterrolebinding temp –clusterrole=cluster-admin –serviceaccount=default:default. It works immediately, which is exactly the problem. Nobody circles back to fix it, and six months later a compromised pod can read every secret in the … Read more

Why Your PodDisruptionBudget Didn’t Save the Last EKS Upgrade

Why Your PodDisruptionBudget Didn’t Save the Last EKS Upgrade

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering A PodDisruptionBudget doesn’t protect your application — it protects a number. That’s why a routine EKS managed node group upgrade recently stalled on us for 40 minutes and paged on-call at 2am, even though our minAvailable: 2 config was textbook. The PDB was doing exactly what … Read more

Your EKS Nodes Run Out of IPs, Not CPU: A Practical Guide to VPC CNI Warming and Prefix Delegation

Your EKS Nodes Run Out of IPs, Not CPU: A Practical Guide to VPC CNI Warming and Prefix Delegation

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering Most engineers size EKS nodes by CPU and memory. We did too — until one day a pod sat in Pending, the node had plenty of compute free, and the events said something about failing to assign an IP. That was our introduction to the quietest … Read more