Learn DevOps, Kubernetes & AWS — Simplified
Practical tutorials, deep dives, and battle-tested guides for engineers who build and ship in the cloud.
By KP | TZoneLabs | DevOps & Cloud Engineering A new model version scored 0.947 against 0.936 for the model in production, passed the promotion gate, ran as a canary with clean dashboards, and got released. Two days later live accuracy had fallen to 0.88. A rolling-count feature was computed over the whole dataset in … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering One GitLab pipeline can run the whole ML lifecycle: test the code and the data, build the training image, run training as a Kubernetes Job, register the model, promote it only if it beats the current one, deploy it as a canary, and release it with … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering A model can return HTTP 200 for months while its predictions get worse. Export three kinds of metrics from the serving pod (latency and errors, the distribution of scores, and the model version), run a scheduled job that compares live input data to the training data … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering Register every trained model in MLflow, promote it with an alias only after CI shows it beats the current one, and deploy by resolving that alias to a fixed version number. Serve it as an ordinary container with a readiness probe, roll it out as a … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering Run training as a Kubernetes Job on a dedicated, tainted GPU node pool. Request nvidia.com/gpu: 1 in the Job, tolerate the pool’s taint, set a deadline so it can’t run forever, and let the autoscaler bring a GPU node up when the Job starts and remove … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering Put the dataset under DVC, point it at an S3 bucket with versioning turned on, and log every training run to MLflow with the data version attached as a tag. That gives a model the same traceability Git gives code: for any model in production, we … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering MLOps is DevOps for a system that ships a trained model alongside the code. The pipelines, registries, rollouts and alerts we already run all carry over. The one thing that changes is that a model can pass every test on Monday and quietly go bad by … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering The kube-prometheus-stack Helm chart gets you Prometheus, Grafana, and Alertmanager running in about ten minutes. What it doesn’t do is scrape your own application’s metrics, write alerting rules that fire on the right thing, or send those alerts anywhere a human will see them. That part … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering Our AWS bill for a new EKS environment came in nine times higher than the month before, and nothing about our traffic had changed. The entire difference was NAT Gateway data processing — a private-subnet VPC with no Gateway Endpoint for S3 was routing every ECR … Read more
By KP | TZoneLabs | DevOps & Cloud Engineering We went looking for a 45-day-old database backup during an incident review and it wasn’t there. Not corrupted, not slow to restore — gone. An S3 lifecycle rule meant to clean up old application logs had been silently deleting backups for over a month, because both … Read more