Learn DevOps, Kubernetes & AWS — Simplified
Practical tutorials, deep dives, and battle-tested guides for engineers who build and ship in the cloud.
By KP | TZoneLabs | DevOps & Cloud Engineering A model can return HTTP 200 for months while its predictions get worse. Export three kinds of metrics from the serving pod (latency and errors, the distribution of scores, and the model version), run a scheduled job that compares live input data to the training data … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering Register every trained model in MLflow, promote it with an alias only after CI shows it beats the current one, and deploy by resolving that alias to a fixed version number. Serve it as an ordinary container with a readiness probe, roll it out as a … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering Run training as a Kubernetes Job on a dedicated, tainted GPU node pool. Request nvidia.com/gpu: 1 in the Job, tolerate the pool’s taint, set a deadline so it can’t run forever, and let the autoscaler bring a GPU node up when the Job starts and remove … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering Put the dataset under DVC, point it at an S3 bucket with versioning turned on, and log every training run to MLflow with the data version attached as a tag. That gives a model the same traceability Git gives code: for any model in production, we … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering MLOps is DevOps for a system that ships a trained model alongside the code. The pipelines, registries, rollouts and alerts we already run all carry over. The one thing that changes is that a model can pass every test on Monday and quietly go bad by … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering The kube-prometheus-stack Helm chart gets you Prometheus, Grafana, and Alertmanager running in about ten minutes. What it doesn’t do is scrape your own application’s metrics, write alerting rules that fire on the right thing, or send those alerts anywhere a human will see them. That part … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering Our AWS bill for a new EKS environment came in nine times higher than the month before, and nothing about our traffic had changed. The entire difference was NAT Gateway data processing — a private-subnet VPC with no Gateway Endpoint for S3 was routing every ECR … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering We went looking for a 45-day-old database backup during an incident review and it wasn’t there. Not corrupted, not slow to restore — gone. An S3 lifecycle rule meant to clean up old application logs had been silently deleting backups for over a month, because both … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering The default Horizontal Pod Autoscaler config scales on raw CPU with no stabilization window, which is exactly why it’s common to see replica counts bounce up and down every few minutes under normal, slightly-uneven traffic. The two settings that actually decide whether HPA is useful or … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering A scheduled quarterly rotation replaced the database credential in Secrets Manager, and within minutes most of the fleet was using it without issue. Two hours later, a subset of pods started failing every query with authentication errors — pods that hadn’t restarted in weeks, still holding … Read more