Our Retrained Model Beat the Champion in CI and Then Got Worse in Production, Because One Feature Peeked Into the Future

By KP | TZoneLabs | DevOps & Cloud Engineering

A new model version scored 0.947 against 0.936 for the model in production, passed the promotion gate, ran as a canary with clean dashboards, and got released. Two days later live accuracy had fallen to 0.88. A rolling-count feature was computed over the whole dataset in training, so it included transactions that happened after each row. In serving it could only see the past. The holdout set had the same flaw, so the gate measured a model that cheated and called it better. We fixed it with one feature definition shared by training and serving, a time-based holdout, and a parity test that fails the pipeline when the two paths disagree.

This is a war story that follows our MLOps series from a DevOps point of view. It leans on the promotion gate from Part 4, the monitoring from Part 5, and the pipeline from Part 6. If the lifecycle is new to you, start with Part 1.

As in the series, each ML concept is lined up against the DevOps task it mirrors, since that is the quickest way to see why the failure was not exotic.

What Maps to What

What Went Wrong	DevOps Equivalent
Training-serving skew	Staging and production built from different config
Feature leakage in the holdout	A test that shares its fixtures with the code it tests
Time-based holdout	Replaying last week’s real traffic against a release
Parity test	A contract test between two services
Canary compared only on errors and latency	A canary that checks 5xx and ignores the business metric
Drift alert in a muted channel	Alert fatigue

What Happened

On a Friday afternoon a pull request added a new feature to our fraud model, a count of how many transactions a customer had made in the last seven days. Fraud tends to arrive in bursts, so the idea was reasonable. The unit tests passed, because they checked that the feature had the right type and no nulls. The merge to the default branch started the pipeline from Part 6.

The timeline, in UTC:

Friday 16:40: PR merged. The pipeline ran the tests, built the training image and started a training Job.
Friday 19:55: training finished. Version 12 scored 0.947 on the holdout, against 0.936 for version 11, the current champion. The gate passed and the canary Deployment rolled out with one pod beside four stable ones.
Monday 10:15: an engineer looked at the canary, saw no errors and normal latency, and approved the manual release job. Version 12 was now on every replica.
Wednesday 09:40: the live accuracy alert fired. Labels for fraud cases arrive about two days after the transaction, so this was the first moment the numbers could show it. Live accuracy was 0.88, against roughly 0.93 before.
Wednesday 10:20: we ran the manual rollback job with version 11. The registry alias followed a few minutes later.

That left about 48 hours of worse decisions on live traffic, and a gate that had said the new model was better.

Where We Started Looking

Step 1: Check whether the pipeline was wrong about the numbers

The first suspicion was our own comparison script. We re-ran it by hand and got the same result. Version 12 really did score 0.947 on that holdout. The gate hadn’t malfunctioned, so the problem was in what it measured.

Step 2: Check the serving pods

Error rate and latency on the stable Deployment were flat across the release. This is the trap from Part 4: a bad model answers quickly and returns 200.

Step 3: Compare the inputs to the training data

The drift dashboard from Part 5 told us something we should have read days earlier. One feature stood out: txn_count_7d had a KS statistic of 0.24 against the training reference, above our 0.2 threshold. No other feature was close. A feature that differs between training data and live data is the definition of skew, so we stopped looking at the model and looked at how the feature was built.

Step 4: Compute the feature both ways on the same rows

The training code looked like this:

df["week"] = df["ts"].dt.to_period("W")
df["txn_count_7d"] = df.groupby(["customer_id", "week"])["txn_id"].transform("count")

The serving code asked the database for a customer’s transactions in the seven days before the request:

def txn_count_7d(customer_id, as_of):
    return db.scalar(
        "SELECT count(*) FROM transactions "
        "WHERE customer_id = %s AND ts >= %s - interval '7 days' AND ts < %s",
        (customer_id, as_of, as_of),
    )

These are not the same feature. The training version counts every transaction in the calendar week that contains the row, including the ones that happen later. The serving version counts only what came before. We recomputed both on a day of logged traffic and the values disagreed on a large share of rows.

Why the Gate Couldn’t See It

The skew explains why production was worse. It doesn’t yet explain why version 12 looked better. Two things combined.

The feature leaked the answer. Fraud arrives in bursts. In the training data, every transaction in a burst got a high count, including the first one, because the count included the rest of the burst. The model learned that a high weekly count means fraud. In production, the first transaction in a burst sees an empty window, which is exactly the one we most needed to catch.

The holdout shared the leak. The gate evaluated on a random 20 percent of the same dataset. Rows from the same bursts landed on both sides of the split, and the leaked feature was computed the same wrong way for both. The holdout rewarded the shortcut, so the gate reported the improvement the shortcut produced.

This is the same shape as a test that builds its input with the same helper as the code it tests: it can’t fail in the way production fails. Version 11 never had the problem because it was trained before this feature existed.

The Fix

Immediate: roll back to the pinned version

Because the Deployment carries a pinned version number, as in Part 4, rolling back was one manual job from the Part 6 pipeline with PREVIOUS_VERSION=11. Then we moved the champion alias back so the registry matched what was running.

Real fix: define each feature once and use it in both places

We replaced the two implementations with one definition that takes the time the prediction is made as a parameter. Training joins it against every historical row. Serving runs it with the time of the incoming request:

SELECT t.txn_id,
       count(h.txn_id) AS txn_count_7d
FROM transactions t
LEFT JOIN transactions h
  ON  h.customer_id = t.customer_id
  AND h.ts >= t.ts - interval '7 days'
  AND h.ts <  t.ts
GROUP BY t.txn_id

The condition h.ts < t.ts is the whole point. A row can only see strictly earlier rows, in training and in serving alike. The self-join is expensive on a large table, so we materialise it into a feature table on a schedule, but the definition lives in one place, in the repository, and both sides read from it.

Evaluate on the future, not on a shuffle

The training script now holds out the most recent week instead of a random sample:

cutoff = df["ts"].max() - pd.Timedelta(days=7)
train = df[df["ts"] <= cutoff]
holdout = df[df["ts"] > cutoff]

This mirrors what production does: the model sees data from after the period it learned from. A leak that depends on rows from the same burst appearing on both sides can’t survive it. The promotion gate from Part 4 now compares the candidate and the champion on this holdout.

What We Put in Place After

1. A parity test that fails the pipeline

Part 5 logs every request’s features and ID. We keep a small sample of that log in the repository and recompute the features through the training path for the same transactions:

def test_training_serving_parity():
    served = pd.read_parquet("tests/data/served_sample.parquet")
    rebuilt = build_training_features(served[["txn_id"]])
    merged = served.merge(rebuilt, on="txn_id", suffixes=("_served", "_train"))

    for col in FEATURE_COLUMNS:
        mismatch = (merged[f"{col}_served"] != merged[f"{col}_train"]).mean()
        assert mismatch < 0.01, f"{col}: {mismatch:.1%} of rows differ"

It runs in the test stage of the Part 6 pipeline, so a feature that disagrees between the two paths stops the merge request before any GPU time is spent. One limit: a brand-new feature has no served history yet. For those, we log the feature in the serving path first, without feeding it to the model, and train on it a week later.

2. A canary comparison we look at before approving

The score histogram from Part 5 already carries the model version. Comparing the mean score of the canary against stable takes one query:

- record: model:mean_score_ratio:canary_vs_stable
  expr: |
    (
      sum(rate(model_score_sum{model_version="12"}[1h]))
      / sum(rate(model_score_count{model_version="12"}[1h]))
    )
    /
    (
      sum(rate(model_score_sum{model_version="11"}[1h]))
      / sum(rate(model_score_count{model_version="11"}[1h]))
    )

In practice the version labels are templated in from the Deployment. A ratio far from 1 doesn’t prove the new model is wrong, since a better model can score differently. It does mean a person has to read the two distributions before pressing the release button, instead of approving because the error rate is flat.

3. Drift alerts that reach someone for new versions

The FeatureDriftHigh alert had fired. It went to a warning channel that was muted, because it also fired every few weeks on a seasonal feature around payday. We now keep the per-feature thresholds Part 5 suggested, and for the first 72 hours after any model release, drift on any feature routes to the on-call engineer. It is the same lesson as our TLS certificate incident: an alert is only useful if it reaches someone who will read it.

What Made This Worse Than It Needed to Be

The canary was judged on HTTP metrics only. It ran for three days and nobody compared its predictions to the stable pods.
The release was approved after a quick look. A manual gate only helps when the reviewer has something to compare.
Labels arrive two days late. Live accuracy is the strongest signal and the slowest. The drift alert was the early warning, and it was muted.
The unit tests checked shape, not meaning. They confirmed the feature existed and had no nulls, and said nothing about when it was computed.
The offline score was trusted on its own. A 1.1 point improvement from one new feature should have prompted a closer look, not a quick merge.

Key Lessons

Define each feature once.
Two implementations of the same feature will eventually disagree, and only production will show it.
Hold out the future, not a shuffle.
A time-based holdout is the offline version of “what happens next week”, and it breaks leaks that a random split rewards.
Test that training and serving agree.
A parity test on logged requests is a contract test between two code paths that produce the same input.
Compare canaries on predictions, not only on errors.
A model can be broken and healthy at the same time, so look at what it returns.
Treat a surprising improvement as a warning.
When a gate says the new model is clearly better after a small change, check why before shipping it.

Summary

What Failed	Why It Went Unnoticed	What We Changed
Feature counted future rows in training	Unit tests checked type and nulls only	One feature definition with an as-of time
Holdout shared the leak	Random split put the same bursts on both sides	Hold out the most recent week
Canary approved on a quick look	Only errors and latency were compared	Mean score ratio checked before release
Drift alert ignored	Muted channel, frequent seasonal noise	Page on-call for 72 hours after a release

None of this was a modelling problem. It was the usual DevOps failure of two environments that look the same and aren’t, with a test that couldn’t tell.