Learn DevOps, Kubernetes & AWS — Simplified
By KP | TZoneLabs | DevOps & Cloud Engineering We went looking for a 45-day-old database backup during an incident review and it wasn’t there. Not corrupted, not slow to restore — gone. An S3 lifecycle rule meant to clean up old application logs had been silently deleting backups for over a month, because both … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering The default Horizontal Pod Autoscaler config scales on raw CPU with no stabilization window, which is exactly why it’s common to see replica counts bounce up and down every few minutes under normal, slightly-uneven traffic. The two settings that actually decide whether HPA is useful or … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering Someone changed a security group in the console to unblock themselves during an incident, and it never made it back into the .tf file. Terraform doesn’t know that happened until the next apply either quietly reverts it or, worse, doesn’t touch it because nothing in the … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering Installing Cluster Autoscaler is one Helm command. Getting it to scale up fast enough and scale down without evicting something it shouldn’t is the part that takes actual configuration. The defaults are conservative on purpose, and left untouched they either leave pending pods waiting longer than … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering Our production API certificate expired on schedule for its routine 60-day renewal, except cert-manager never got a new one issued. The cause had nothing to do with that certificate. A staging Ingress with a typo’d Route53 zone ID had been failing its own renewal silently for … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering A single main.tf with every resource for every environment works for the first few weeks of a project. It stops working the day you need a second environment, and by the time a team is running terraform plan against a 2,000-line file, nobody wants to touch … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering Every terraform plan across three separate pipelines started failing with the same message: Error acquiring the state lock. Nothing was running. The lock had been held by a CI job that got killed by a pipeline timeout forty minutes earlier, and it never got the chance … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering During a traffic spike, every app server started throwing the same generic message: ETIMEDOUT. Nothing named the database. Nothing named the connection pool. So for the first hour, we debugged the network, the load balancer, and the application code, in that order, before anyone checked how … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering For nine days, our cluster added and removed the same three worker nodes every six minutes. Every dashboard we watched stayed green the whole time. Pods were running, nodes were Ready, CPU never crossed 60%. Nobody noticed until the AWS bill landed about 40% higher than … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering An AccessDenied error from AWS tells you almost nothing. It names the action and sometimes the resource, but never which of the five places a permission could be blocked actually blocked it. We used to open the IAM console and read policies line by line, sometimes … Read more