What Is MLOps? The ML Lifecycle Explained for DevOps Engineers

By KP | TZoneLabs | DevOps & Cloud Engineering

MLOps is DevOps for a system that ships a trained model alongside the code. The pipelines, registries, rollouts and alerts we already run all carry over. The one thing that changes is that a model can pass every test on Monday and quietly go bad by the end of the month, with no commit in between.

This is Part 1 of a short series on MLOps from a DevOps point of view. We skip the data science and stick to the parts that land on an infrastructure team: what the lifecycle looks like, which piece maps to which DevOps habit, and where the operational pain tends to show up first.

Each MLOps concept below is lined up against the DevOps task it mirrors, since reasoning from something you already run day to day is the fastest way into an unfamiliar workflow.

Why a Model Isn’t Just Another Build Artifact

A normal build is a function of source code. Same commit, same artifact. A trained model is a function of three things: the code, the training data, and the settings used to run training (hyperparameters, random seed, library versions). Change any one of them and the artifact changes, even though nobody touched the repo.

The second difference is that a model keeps changing after it ships. A web service that returned correct answers last week still returns correct answers this week. A fraud model trained on last quarter’s transactions slowly gets worse as customer behaviour shifts. Nothing crashes and nothing logs an error. The HTTP status is 200 the whole time.

Most of MLOps exists to deal with those two facts: the artifact has more inputs than code, and “deployed and healthy” doesn’t stay true on its own.

The ML Lifecycle, Mapped to What We Already Run

Each stage has a DevOps equivalent. Seeing it that way makes the tooling much less foreign.

1. Data versioning (source control for datasets)

If the data is an input to the build, it needs a version. Tools like DVC and lakeFS keep a small pointer file in Git while the actual data lives in object storage. The commit then records both which code and which data produced a given model. The cheap starting point is an S3 bucket with versioning turned on and the dataset path pinned in the training config.

2. Training pipeline (CI for models)

Training is a containerised job that runs on a trigger, a schedule, or a new batch of data. It reads a pinned data version, produces a model file and a metrics file, and exits. The orchestration can be Argo Workflows, Kubeflow Pipelines, or a plain GitLab CI job that launches a Kubernetes Job. The infra differences from normal CI are GPU nodes, long runtimes and large inputs.

3. Experiment tracking and model registry (artifact registry)

Every training run records its parameters, metrics and output. The model registry holds the versions that passed review, much like a container registry holds images. MLflow is the common open-source choice. A registered model carries a version number and a stage such as staging or production, plus a link back to the code commit and data version that created it.

import mlflow

mlflow.register_model(
    model_uri=f"runs:/{run_id}/model",
    name="fraud-detector",
)

4. Serving (deployment)

A model is served like any other service: packaged in a container, exposed over REST or gRPC, and put behind a Deployment, Service and autoscaler. KServe, Seldon and BentoML wrap this for you. A hand-rolled FastAPI app with the model file loaded at start-up works fine for a first version. Canary rollouts matter even more here, because a new model version can look healthy at the HTTP level while giving worse predictions.

5. Monitoring (observability)

Latency, error rate and saturation still apply, and the same Prometheus and Grafana setup covers them. On top of that sit model-specific signals: whether the input data still looks like the training data (data drift), whether the distribution of predictions has shifted, and whether accuracy holds up once real outcomes arrive, which for some models takes days or weeks.

6. Retraining (the loop CD never had)

When drift crosses a threshold, or on a fixed schedule, the pipeline in step 2 runs again and the new model goes back through the registry and rollout. Closing this loop is what separates a model that was deployed once from a system that is actually operated.

A Minimal Pipeline Sketch

The shape is a normal CI pipeline with two extra gates. Data is validated before training starts, and an evaluation step fails the pipeline if the metrics drop under a threshold, the same way a failing test blocks a merge:

stages: [validate-data, train, evaluate, register, deploy]

validate-data:
  stage: validate-data
  script:
    - python validate_data.py --version $DATA_VERSION

train:
  stage: train
  script:
    - python train.py --data-version $DATA_VERSION --out model.pkl
  artifacts:
    paths: [model.pkl, metrics.json]

evaluate:
  stage: evaluate
  script:
    - python evaluate.py --metrics metrics.json --min-accuracy 0.90

The scripts are placeholders, but the structure is the point. Later parts of the series fill in each stage with real tooling.

Where an Infrastructure Team Gets Pulled In

Data scientists tend to own the model code. The parts that end up on the platform team are the ones we already know how to run:

GPU node pools. Separate node groups, taints and tolerations, and autoscaling that understands expensive instances. Defining them in Terraform modules with remote state keeps them reproducible.
Large artifacts. A model image or weights file can be several gigabytes, which changes image pull times and how fast a new pod becomes ready.
Access to data. Training jobs need read access to datasets and write access to the registry. Scoping that without opening up whole buckets is a familiar IAM problem, and debugging IAM denials works the same way here.
Reproducible environments. Pinned base images and library versions, so a model trained today can be rebuilt next year.

Mistakes That Show Up Later, Not Immediately

Training in a notebook with no recorded data version. The model works, and nobody can say what it was trained on or rebuild it.
Deploying with no evaluation gate. Without a metric threshold in the pipeline, a bad training run reaches production as easily as a good one.
Watching only latency and 5xx. A model that returns confident nonsense still returns 200.
Loading weights from a latest tag at pod start. It’s the same problem as an unpinned module source: a restart can change behaviour with no change on your side.
No path back to retraining. A model that can only be shipped once is a snapshot, and it starts aging on day one.

Key Lessons

A model has more inputs than code.
Data and training settings are part of the build, so both need versions that get recorded next to the artifact.
Every DevOps stage has an ML equivalent.
Source control maps to data versioning, CI to training pipelines, the artifact registry to the model registry, and observability to drift monitoring.
Put an evaluation gate in the pipeline.
A metric threshold that fails the build does for a model what a failing test does for code.
Healthy and correct are different checks.
Standard SLIs tell you the service is up. Drift and prediction metrics tell you whether it is still right.
Plan the retraining loop early.
Retrofitting it after the first model has been in production for months is much harder than building it in.

Summary

ML Stage	DevOps Equivalent	Common Tooling
Data versioning	Source control	DVC, lakeFS, versioned S3
Training pipeline	CI	Argo Workflows, Kubeflow, GitLab CI
Model registry	Artifact registry	MLflow
Serving	Deployment and rollout	KServe, Seldon, BentoML
Monitoring	Observability	Prometheus, Grafana, drift checks
Retraining	Feedback loop into CI	Scheduled or drift-triggered pipelines

If you already run pipelines, registries and dashboards, you know most of MLOps. The rest is learning where the data and the model add new things to version and watch.