How to Monitor ML Models in Production: Drift Detection With Prometheus and Grafana

By KP | TZoneLabs | DevOps & Cloud Engineering

A model can return HTTP 200 for months while its predictions get worse. Export three kinds of metrics from the serving pod (latency and errors, the distribution of scores, and the model version), run a scheduled job that compares live input data to the training data and pushes a drift number to Prometheus, and alert on that number the way we alert on CPU. When it fires, start the same training pipeline we already built, and let the promotion gate decide whether the new model ships.

This is Part 5, the last part of our MLOps series from a DevOps point of view. Part 1 mapped the lifecycle, Part 2 versioned data and experiments, Part 3 ran training on GPU nodes, and Part 4 promoted and served models. This one closes the loop with monitoring and retraining.

Each concept below is lined up against the DevOps task it mirrors, since reasoning from something you already run day to day is the fastest way into an unfamiliar workflow.

What Maps to What

Model Monitoring Task	DevOps Equivalent
Latency and error metrics on the serving pod	The usual golden signals for any service
Score distribution	A business metric such as orders per minute
Data drift check	A config drift check comparing live state to what the code says
Scheduled drift job	A synthetic check or a nightly audit job
Live accuracy from delayed outcomes	An end-to-end check that only completes after the real result arrives
Alert that triggers retraining	An alert that fires a runbook automation

Step 1: Export Serving Metrics From the Model API

Start with the FastAPI service from Part 4. The standard Prometheus client adds a request counter, a latency histogram and a histogram of the scores the model returns. Every metric carries the model version, so a rollout shows up on the graph:

import os
import time
from prometheus_client import Counter, Histogram, make_asgi_app

MODEL_VERSION = os.environ["MODEL_URI"].rsplit("/", 1)[-1]

PREDICTIONS = Counter(
    "model_predictions_total", "Predictions served", ["model_version"]
)
LATENCY = Histogram(
    "model_predict_seconds", "Time spent in predict", ["model_version"]
)
SCORES = Histogram(
    "model_score", "Distribution of returned scores", ["model_version"],
    buckets=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9],
)

app.mount("/metrics", make_asgi_app())

@app.post("/predict")
def predict(payload: dict):
    rows = pd.DataFrame(payload["rows"])
    start = time.perf_counter()
    scores = model.predict(rows).tolist()
    LATENCY.labels(MODEL_VERSION).observe(time.perf_counter() - start)
    for s in scores:
        SCORES.labels(MODEL_VERSION).observe(s)
    PREDICTIONS.labels(MODEL_VERSION).inc(len(scores))
    return {"model": MODEL_URI, "predictions": scores}

This assumes the model returns a score between 0 and 1. Adjust the buckets if yours returns something else. The score histogram is the cheapest early warning there is. If the share of scores above 0.9 doubles overnight, something changed, even though no request failed. Our Prometheus and Grafana setup post covers scraping a new target and building the dashboard.

Step 2: Keep a Reference Sample From Training

Drift only means something relative to a baseline. At the end of every training run, save a sample of the training features next to the model, and version it with the same DVC flow from Part 2:

train_df[feature_columns].sample(n=50000, random_state=42).to_parquet(
    "s3://acme-ml-data/reference/train_features.parquet"
)

This plays the part of the “desired state” in a config drift check. The live inputs are compared to it.

Step 3: Log What the Model Sees

The drift job needs recent inputs. Log each request’s features and an ID to a place the job can read, such as a daily Parquet file in S3. A log line per prediction with the model version, the request ID and the input features is enough. Check what the features contain before you do this, since it may include personal data that your retention rules apply to.

Step 4: Run a Scheduled Drift Check

A CronJob compares the last day of live inputs to the reference sample, one statistical test per feature. The two-sample Kolmogorov-Smirnov test works for numeric features, and the statistic it returns is a number from 0 (identical distributions) to 1 (completely different):

import pandas as pd
from scipy.stats import ks_2samp
from prometheus_client import CollectorRegistry, Gauge, push_to_gateway

reference = pd.read_parquet("s3://acme-ml-data/reference/train_features.parquet")
recent = pd.read_parquet("s3://acme-ml-data/inference-logs/last-24h.parquet")

registry = CollectorRegistry()
drift = Gauge(
    "feature_drift_ks_statistic",
    "KS statistic between training and live data",
    ["feature"],
    registry=registry,
)

for col in reference.columns:
    stat, _ = ks_2samp(reference[col], recent[col])
    drift.labels(feature=col).set(stat)

push_to_gateway("pushgateway.monitoring.svc:9091", job="drift-check", registry=registry)

apiVersion: batch/v1
kind: CronJob
metadata:
  name: drift-check
spec:
  schedule: "0 6 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      activeDeadlineSeconds: 1800
      template:
        spec:
          restartPolicy: Never
          serviceAccountName: ml-monitoring
          containers:
            - name: drift
              image: registry.example.com/ml/drift-check:1.0.0
              resources:
                requests:
                  cpu: "1"
                  memory: 4Gi

A batch job is the right shape here because a daily number is enough for most models, and it keeps this work off the serving pods. The Pushgateway is needed because a short-lived job can’t be scraped. It’s a normal Deployment, and it is the one place in Prometheus where pushing is the accepted pattern.

Step 5: Alert on Drift and on a Silent Checker

This needs the Prometheus Operator for the PrometheusRule resource. If you run plain Prometheus, the same expressions go in a rules file:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: model-drift
spec:
  groups:
    - name: model-drift
      rules:
        - alert: FeatureDriftHigh
          expr: max(feature_drift_ks_statistic) > 0.2
          for: 1h
          labels:
            severity: warning
          annotations:
            summary: "Live input data has drifted from the training data"
        - alert: DriftCheckNotRunning
          expr: time() - push_time_seconds{job="drift-check"} > 129600
          labels:
            severity: warning
          annotations:
            summary: "The drift check has not pushed metrics in over 36 hours"

The second rule matters as much as the first. Pushgateway keeps the last value it received forever, so a broken CronJob leaves the old, healthy-looking number on the dashboard. Alerting on the push time catches that. It’s the same lesson as our TLS certificate incident: an alert only helps if something is also watching that the thing producing the data still works.

The 0.2 threshold is a starting point. The KS statistic gets more sensitive as the sample grows, so with large samples it flags changes that don’t hurt the model. Watch the numbers for a couple of weeks and set the threshold above normal day-to-day variation.

Step 6: Measure Real Accuracy When the Answers Arrive

Drift says the inputs changed, not that the model got worse. Accuracy is the real signal, but for many models the true outcome shows up days later, such as a chargeback or a confirmed fraud case. Log the request ID with each prediction, join it to the outcomes table when the labels arrive, and push the result as one more gauge:

accuracy = (joined["prediction"] == joined["actual"]).mean()
live_accuracy = Gauge("model_live_accuracy", "Accuracy on labelled live traffic", registry=registry)
live_accuracy.set(accuracy)

Alert on a drop against the accuracy recorded at training time, which MLflow already holds from Part 2. Because the labels lag, this alert is slow, and drift is the early signal that covers the gap.

Step 7: Close the Loop With Retraining

When a drift or accuracy alert fires, Alertmanager can call a webhook that starts the training pipeline. With GitLab, that is the pipeline trigger API:

curl -X POST \
  -F token=$TRIGGER_TOKEN \
  -F ref=main \
  https://gitlab.example.com/api/v4/projects/$PROJECT_ID/trigger/pipeline

The retrained model goes through everything from Parts 2 to 4: versioned data, a GPU Job, registration, the comparison gate against the champion, a pinned deploy and a canary. Nothing about the alert skips the gate. If the new model isn’t better than the current one, the pipeline stops and a person looks at it. Starting a retrain automatically is fine. Shipping one automatically is a choice to make deliberately, with a canary behind it.

Mistakes That Show Up Later, Not Immediately

Watching latency and 5xx only. That is the setup that missed the problem in the first place.
Drift alerts with no reference sample. A baseline taken from last week’s traffic hides slow drift. Compare against the training data.
A stale Pushgateway value. The dashboard looks healthy because nothing has pushed in a month. Alert on the push time.
One global drift threshold. A feature that varies naturally by day of week will fire every Monday. Look at per-feature history before choosing a number.
Auto-retrain wired straight to auto-deploy. A bad batch of data produces a bad model and ships it. Keep the comparison gate and the canary.
No model version label on metrics. After a rollout, nobody can tell whether a change came from the new model or from the data.
Logging raw inputs with no retention plan. The inference log becomes a store of personal data nobody owns.

Key Lessons

Healthy and correct are different checks.
Request metrics say the service is up. Score distributions, drift and live accuracy say whether it is still right.
Compare live data to the training data.
Keep a reference sample with every model so drift has something fixed to be measured against.
Alert on the checker as well as the result.
A silent drift job looks identical to a healthy model unless something watches its last run time.
Use drift as the early warning and accuracy as the proof.
Labels arrive late, so the two signals cover different parts of the timeline.
Automate the retrain, gate the release.
The alert can start the pipeline, but the same comparison and canary decide what ships.

Summary

Signal	Source	Speed
Latency and errors	Serving pod metrics	Seconds
Score distribution	Histogram in the API	Minutes to hours
Input drift	CronJob with KS test, via Pushgateway	Daily
Live accuracy	Predictions joined to later outcomes	Days, depending on label delay

The Whole Series in One Table

Part	MLOps Stage	DevOps Equivalent
1	The lifecycle	The CI/CD and ops stages we already run
2	Data and experiment versioning	Source control, lockfiles, build logs
3	Training on GPU nodes	One-shot CI jobs on tagged runners
4	Registry, promotion and serving	Image registry, release gates, canaries
5	Monitoring, drift and retraining	Observability, drift checks, runbook automation

None of these stages needed a new way of thinking about operations. Each one needed one new thing to version or watch, and the rest was infrastructure we already know how to run.