Our Retrained Model Beat the Champion in CI and Then Got Worse in Production, Because One Feature Peeked Into the Future
By KP | TZoneLabs | DevOps & Cloud Engineering A new model version scored 0.947 against 0.936 for the model in production, passed the promotion gate, ran as a canary with clean dashboards, and got released. Two days later live accuracy had fallen to 0.88. A rolling-count feature was computed over the whole dataset in … Read more