Learn DevOps, Kubernetes & AWS — Simplified
Practical tutorials, deep dives, and battle-tested guides for engineers who build and ship in the cloud.
By KP | TZoneLabs | DevOps & Cloud Engineering Every deploy of one gRPC service left about 5% of calls failing for the next 30 to 40 seconds, then it cleared up on its own. The rollout itself looked clean, every new pod passed its readiness probe on schedule. The failing calls were connecting to … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering Installing Cluster Autoscaler is one Helm command. Getting it to scale up fast enough and scale down without evicting something it shouldn’t is the part that takes actual configuration. The defaults are conservative on purpose, and left untouched they either leave pending pods waiting longer than … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering Our production API certificate expired on schedule for its routine 60-day renewal, except cert-manager never got a new one issued. The cause had nothing to do with that certificate. A staging Ingress with a typo’d Route53 zone ID had been failing its own renewal silently for … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering A single main.tf with every resource for every environment works for the first few weeks of a project. It stops working the day you need a second environment, and by the time a team is running terraform plan against a 2,000-line file, nobody wants to touch … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering Every terraform plan across three separate pipelines started failing with the same message: Error acquiring the state lock. Nothing was running. The lock had been held by a CI job that got killed by a pipeline timeout forty minutes earlier, and it never got the chance … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering Registering a GitLab Runner with gitlab-runner register and walking away is how you end up with three untagged runners fighting over the same jobs, a deploy job that occasionally runs on whatever box happens to be free, and a host that falls over the moment four … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering During a traffic spike, every app server started throwing the same generic message: ETIMEDOUT. Nothing named the database. Nothing named the connection pool. So for the first hour, we debugged the network, the load balancer, and the application code, in that order, before anyone checked how … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering Every rolling deploy for two months sent a burst of 502 errors to a small percentage of users, for about 30 seconds each time. Support tickets blamed “the app being slow sometimes.” The actual cause: our readiness probe was checking a port the container wasn’t listening … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering For three weeks, our staging database password was sitting in plain text in every CI job log a specific pipeline produced. Anyone with read access to the project, which included a few external contractors, could open a job log and copy it out. Nobody stole it, … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering The default .gitlab-ci.yml most teams start with runs npm ci or pip install in every single job, on every single commit. A pipeline with a build, test, and lint job doing the same install three times isn’t three times safer. It’s three times slower for no … Read more