● DevOps · Kubernetes · AWS

Master Cloud Engineering
From Real-World Experience

Practical tutorials, deep dives, and battle-tested guides for engineers who build and ship in the cloud.

Browse by topic → AWS Kubernetes DevOps GitLab

Setting Up Kubernetes RBAC: The Roles and RoleBindings That Actually Enforce Least Privilege

Setting Up Kubernetes RBAC: The Roles and RoleBindings That Actually Enforce Least Privilege

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering A Role scoped to one namespace, bound to one service account, with verbs limited to exactly what that workload does, is what turns Kubernetes RBAC into an actual security boundary instead of a checkbox. Most clusters we’ve audited get this half right: the Role exists, but … Read more

A Namespace Relabel Silently Broke a NetworkPolicy Selector, and Cross-Namespace Traffic Started Timing Out

A Namespace Relabel Silently Broke a NetworkPolicy Selector, and Cross-Namespace Traffic Started Timing Out

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering A payments-team service started timing out on calls to a shared internal API, intermittently, with nothing in either side’s logs pointing at why. No connection refused, no TLS error, no 5xx. Just a hang until the client’s own timeout gave up. The pods were healthy, the … Read more

Spinning Up Per-Merge-Request Review Environments With GitLab CI Dynamic Environments

Spinning Up Per-Merge-Request Review Environments With GitLab CI Dynamic Environments

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering A dynamic environment: block in .gitlab-ci.yml turns every merge request into its own deployed, reachable review app, and turns every closed merge request into a cleanup job that tears it back down. Reviewers stop asking “can you deploy this so I can look at it” and … Read more

A CoreDNS Cache Setting Kept Routing gRPC Traffic to Pods That No Longer Existed After Every Deploy

A CoreDNS Cache Setting Kept Routing gRPC Traffic to Pods That No Longer Existed After Every Deploy

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering Every deploy of one gRPC service left about 5% of calls failing for the next 30 to 40 seconds, then it cleared up on its own. The rollout itself looked clean, every new pod passed its readiness probe on schedule. The failing calls were connecting to … Read more

Setting Up Kubernetes Cluster Autoscaler on EKS: The Settings That Actually Matter

Setting Up Kubernetes Cluster Autoscaler on EKS: The Settings That Actually Matter

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering Installing Cluster Autoscaler is one Helm command. Getting it to scale up fast enough and scale down without evicting something it shouldn’t is the part that takes actual configuration. The defaults are conservative on purpose, and left untouched they either leave pending pods waiting longer than … Read more

A Typo’d Route53 Zone ID Quietly Exhausted Our ACME Rate Limit, and It Took Down an Unrelated Production Certificate

A Typo’d Route53 Zone ID Quietly Exhausted Our ACME Rate Limit, and It Took Down an Unrelated Production Certificate

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering Our production API certificate expired on schedule for its routine 60-day renewal, except cert-manager never got a new one issued. The cause had nothing to do with that certificate. A staging Ingress with a typo’d Route53 zone ID had been failing its own renewal silently for … Read more

Structuring Terraform Modules and Remote State: What Actually Scales Past One Environment

Structuring Terraform Modules and Remote State: What Actually Scales Past One Environment

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering A single main.tf with every resource for every environment works for the first few weeks of a project. It stops working the day you need a second environment, and by the time a team is running terraform plan against a 2,000-line file, nobody wants to touch … Read more

A Killed CI Job Left a Terraform State Lock Behind, and It Blocked Every Pipeline for 40 Minutes

A Killed CI Job Left a Terraform State Lock Behind, and It Blocked Every Pipeline for 40 Minutes

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering Every terraform plan across three separate pipelines started failing with the same message: Error acquiring the state lock. Nothing was running. The lock had been held by a CI job that got killed by a pipeline timeout forty minutes earlier, and it never got the chance … Read more

Setting Up Self-Hosted GitLab Runners: Tags, Scaling, and the Config Mistakes That Waste Compute

Setting Up Self-Hosted GitLab Runners: Tags, Scaling, and the Config Mistakes That Waste Compute

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering Registering a GitLab Runner with gitlab-runner register and walking away is how you end up with three untagged runners fighting over the same jobs, a deploy job that occasionally runs on whatever box happens to be free, and a host that falls over the moment four … Read more

A Traffic Spike Exhausted Our Database Connection Pool, and Generic Timeout Errors Sent Us Debugging the Wrong Layer for an Hour

A Traffic Spike Exhausted Our Database Connection Pool, and Generic Timeout Errors Sent Us Debugging the Wrong Layer for an Hour

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering During a traffic spike, every app server started throwing the same generic message: ETIMEDOUT. Nothing named the database. Nothing named the connection pool. So for the first hour, we debugged the network, the load balancer, and the application code, in that order, before anyone checked how … Read more