How to Promote and Serve Models From an MLflow Registry on Kubernetes: A DevOps Engineer's Guide - TZoneLabs How to Promote and Serve Models From an MLflow Registry on Kubernetes: A DevOps Engineer's Guide

By KP | TZoneLabs | DevOps & Cloud Engineering

Register every trained model in MLflow, promote it with an alias only after CI shows it beats the current one, and deploy by resolving that alias to a fixed version number. Serve it as an ordinary container with a readiness probe, roll it out as a canary, and roll back by redeploying the previous version number. The alias is for people and CI. The Deployment should always carry a pinned version.

This is Part 4 of our MLOps series from a DevOps point of view. Part 1 mapped the ML lifecycle onto the DevOps stages we already run, Part 2 covered versioning data and experiments, and Part 3 ran training on GPU nodes. This one covers what happens after a training run finishes: the registry and serving.

Each concept below is lined up against the DevOps task it mirrors, since reasoning from something you already run day to day is the fastest way into an unfamiliar workflow.

What Maps to What

Model Task	DevOps Equivalent
MLflow model registry	A container image registry
Model version number	An image digest
Alias such as `champion`	A moving tag such as `stable`
Promotion check in CI	A test gate before a release
Canary model rollout	A canary deploy of a service
Redeploy the previous version	`kubectl rollout undo`

Step 1: Register the Model From the Training Run

At the end of training, register the logged model. Each call creates a new numbered version under the same model name, the way each push creates a new image under the same repository:

import mlflow

result = mlflow.register_model(
    model_uri=f"runs:/{run_id}/model",
    name="fraud-detector",
)
print(result.version)

The version carries a link back to the run, and the run carries the commit and data hash from Part 2. A model in production can be traced all the way back to its inputs.

Step 2: Promote With an Alias, but Only After a Comparison

Part 1 mentioned registry stages. Recent MLflow versions steer toward aliases instead, so that is what we use here. An alias is a named pointer to one version, and moving it is the promotion step. The gate is a short script in the pipeline that refuses to move the alias unless the candidate beats the current champion:

import sys
from mlflow import MlflowClient
from mlflow.exceptions import MlflowException

NAME = "fraud-detector"
candidate_version = sys.argv[1]
client = MlflowClient()

def accuracy(version):
    return client.get_run(version.run_id).data.metrics["accuracy"]

candidate = client.get_model_version(NAME, candidate_version)

try:
    champion = client.get_model_version_by_alias(NAME, "champion")
    if accuracy(candidate) < accuracy(champion):
        sys.exit(f"Candidate v{candidate_version} is worse than champion v{champion.version}")
except MlflowException:
    pass  # no champion yet, first model gets promoted

client.set_registered_model_alias(NAME, "champion", candidate_version)

This is the same idea as the evaluation gate from Part 1, now applied against the model currently in production and not just a fixed threshold. A bad training run stops here with a failed job and a clear message.

Step 3: Resolve the Alias to a Pinned Version Before Deploying

This is the step that is easy to skip. If a pod loads models:/fraud-detector@champion when it starts, then every pod restart can silently change the model, and two replicas started an hour apart can serve different versions. It is the same trap as deploying an image by a moving tag. Resolve the alias once in CI and write the number into the manifest:

from mlflow import MlflowClient

print(MlflowClient().get_model_version_by_alias("fraud-detector", "champion").version)

deploy:
  stage: deploy
  script:
    - export MODEL_VERSION=$(python resolve_version.py)
    - sed "s/MODEL_VERSION/$MODEL_VERSION/g" k8s/fraud-detector.yaml | kubectl apply -f -

Step 4: Serve the Model as a Normal Service

A small FastAPI app is enough for a first version. It loads the model once at start-up and reports which version it is serving:

import os
import mlflow.pyfunc
import pandas as pd
from fastapi import FastAPI

MODEL_URI = os.environ["MODEL_URI"]
model = mlflow.pyfunc.load_model(MODEL_URI)
app = FastAPI()

@app.get("/healthz")
def healthz():
    return {"status": "ok", "model": MODEL_URI}

@app.post("/predict")
def predict(payload: dict):
    rows = pd.DataFrame(payload["rows"])
    return {"model": MODEL_URI, "predictions": model.predict(rows).tolist()}

Returning the model URI in every response is a small habit that pays off during an incident. When someone asks which model produced a strange prediction, the answer is in the response itself.

Step 5: Deploy It With a Readiness Probe

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fraud-detector-stable
spec:
  replicas: 4
  selector:
    matchLabels:
      app: fraud-detector
      track: stable
  strategy:
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    metadata:
      labels:
        app: fraud-detector
        track: stable
    spec:
      serviceAccountName: ml-serving
      containers:
        - name: api
          image: registry.example.com/ml/fraud-detector-api:1.4.0
          env:
            - name: MODEL_URI
              value: models:/fraud-detector/MODEL_VERSION
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 20
            periodSeconds: 5
          resources:
            requests:
              cpu: "1"
              memory: 2Gi

The model loads before the server starts accepting requests, so the readiness probe only passes once the weights are in memory. With maxUnavailable: 0, a rolling update never removes a working pod until its replacement is ready. If the serving pods need to pull the model from S3, give them a scoped role the way Part 3 did for training, and use the IAM debugging steps if a load fails with an access error.

Step 6: Roll Out a New Model as a Canary

A second Deployment with track: canary and the same app: fraud-detector label sits behind the same Service. The Service selects only on app, so traffic splits by replica count. One canary pod next to four stable pods is roughly 20 percent of requests:

apiVersion: v1
kind: Service
metadata:
  name: fraud-detector
spec:
  selector:
    app: fraud-detector
  ports:
    - port: 80
      targetPort: 8080

The canary Deployment is a copy of the stable one with track: canary, replicas: 1 and the candidate’s MODEL_URI. A replica ratio is a rough split, and tools like Argo Rollouts and KServe add exact weighted traffic if you need it. What matters more is what you watch. HTTP errors will look fine for a bad model, so compare prediction distributions and business metrics between the two tracks. Part 5 covers that in detail.

If the canary holds up, deploy the same version to the stable Deployment and delete the canary one.

Step 7: Roll Back

Because the version number lives in the manifest, rolling back is an ordinary redeploy of the previous number:

kubectl set env deployment/fraud-detector-stable \
  MODEL_URI=models:/fraud-detector/6

Then move the champion alias back to version 6 so the registry matches what is running. If the manifest is also in Git, a revert of the last commit does the same job with a review trail.

Mistakes That Show Up Later, Not Immediately

Loading the alias at pod start. A restart or a scale-up can pick up a new model nobody deployed on purpose. Pin the number.
Promoting without comparing to the current model. A fixed threshold lets a worse model replace a better one as long as it clears the bar.
No model version in responses or logs. When a prediction looks wrong, nobody can say which model made it.
Judging a canary by HTTP errors alone. A bad model answers quickly and returns 200.
Large weights downloaded on every pod start. Rollouts slow down and a registry outage stops new pods from starting. Bake the model into the image, or cache it on a volume, for big models.
Letting any branch move the champion alias. Restrict who and which pipelines can promote, the same way production deploys are restricted.

Key Lessons

Treat the registry like an image registry.
Versions are immutable, and aliases are moving pointers for humans and CI.
Gate promotion on a comparison with the current model.
Beating the champion is a stronger check than clearing a fixed threshold.
Deploy a pinned version, never an alias.
Resolve the alias once in CI so every replica serves the same model.
Make the serving pod report its model.
The version in each response turns a vague complaint into a specific answer.
Keep rollback boring.
When the version is in the manifest, going back is one command or one revert.

Summary

Stage	Action	Key Call or Setting
Register	New version per training run	`register_model`
Promote	Compare, then move the alias	`set_registered_model_alias`
Deploy	Resolve the alias to a number in CI	`MODEL_URI` with a version
Release	Canary Deployment behind one Service	`track` label, replica ratio
Roll back	Redeploy the previous number	`kubectl set env` or a Git revert

The one change worth making this week is pinning the version in the Deployment. The rest of the list builds on that.

How to Promote and Serve Models From an MLflow Registry on Kubernetes: A DevOps Engineer’s Guide