Skip to main content
DevOps Insightsguides & answers

MLOps · troubleshooting

The model performs worse in production than in evaluation

Evaluation said one thing and production says another, yet nothing errors, nothing restarts, and the service is healthy. That is why this is usually found late and by somebody outside engineering.

First published

New to this? Read what is actually happening first, then come back to the steps.

On this page · 9 sections

Before you start

  • Logs of what the model received in production: the feature values it actually scored, with the model version and a request ID, not only the raw request and the prediction.
  • The training and evaluation pipeline code, and a snapshot of the data the evaluation used, including how it was split into training and test sets.
  • The deployed model version with its preprocessing artifacts, and the package list of the serving environment (pip freeze run inside it).
  • The live metric that showed the drop, with dates, so you can see whether the gap was there from the first day or grew.
  • Approval to handle production data in your analysis environment, since every comparison here copies real inputs out of serving.

Run this first

step 1 of 5
# Was the gap there on day one, or did it grow?

Look for The shape of the metric since deployment, not its current value

This single question separates an engineering bug from drift, and they share no fixes. A gap present from the first request is a pipeline or evaluation problem; one that grew is the world moving.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

Was the gap present from the first day of serving, or did it grow over time?

This is the highest-value question available and it splits the causes almost cleanly.

Cause space

6 of 6 still possible

  • A feature is computed differently in training and servingCommon
  • The features used at serving time were never loggedCommon
  • The input data has changed since trainingCommon
  • Evaluation was optimistic because of leakageOccasional
  • The serving stack runs different code or versionsOccasional
  • The model is shaping the data it later learns fromRare

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Or diagnose it manually

In this order. The first command usually contains the whole answer.

  1. Step 1

    # Was the gap there on day one, or did it grow?

    This single question separates an engineering bug from drift, and they share no fixes. A gap present from the first request is a pipeline or evaluation problem; one that grew is the world moving.

    Look for The shape of the metric since deployment, not its current value

  2. Step 2

    # Score one identical production example through the training path and the serving path

    The most direct test available. If the feature vectors differ, you have found it and nothing else needs investigating.

    Look for Any feature that differs between the two

  3. Step 3

    # Compare live input distributions against the training set, per feature

    Needs no labels and no ground truth, so it works immediately and on any model. Catches drift and several format changes.

    Look for A null rate, category set or mean that has moved since training

  4. Step 4

    # Re-evaluate with a time-based split instead of a random one

    Rules out the possibility that production is fine and the original evaluation was optimistic.

    Look for Offline scores falling toward the production number

  5. Step 5

    pip freeze  # in the serving image, diffed against the training environment

    Cheap, and catches the class where the model is identical and its preprocessing is not.

    Look for Different versions of anything that transforms input, or parameters recomputed at serving

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

A feature is computed differently in training and serving

Common

The same named feature is produced by two pieces of code — an offline batch job and an online request path — and they do not agree. Different rounding, a different time window, refunds included in one and not the other, a null handled as zero in one and dropped in the other. The model was trained on one definition and is asked about the other on every single request.

Confirm

# Score the SAME production example through both paths and diff the feature vectors
# (no CLI for this — it is a comparison you have to arrange)

Any feature whose value differs between the offline and online computation for one identical input. One mismatched feature is enough to explain a large gap

Fix

  • Compute the feature once, in one place, and have both training and serving read that. This is one of the main reasons feature stores exist (Google's MLOps guide also lists reusing feature sets), and the benefit comes from the single definition rather than from buying a product.
  • Where you cannot unify immediately, add an automated comparison that fails the pipeline when the two implementations disagree on a sample of real inputs.
  • Treat a feature definition as an interface with one owner, not as code that happens to appear twice.

The features used at serving time were never logged

Common

Not a cause of the gap so much as the reason you cannot find it. If only the model's inputs at training time are recorded, there is nothing to compare against. Google's guidance is direct about this: save the set of features used at serving time, and pipe those to a log to use at training time.

Confirm

# Check whether your serving path records the feature vector it scored,
# not merely the request and the prediction

Absence of the served feature vector. If it is missing, this is the first thing to fix — every other diagnosis below depends on it

Fix

  • Log the exact feature vector that was scored, with the model version and a request identifier.
  • Sample rather than logging everything if volume is a concern; a consistent sample is enough for distribution comparison.
  • Then train from those logged features where you can, which removes this class of skew by construction.

The input data has changed since training

Common

The pipeline is correct and the world moved. Traffic shifted to mobile, a customer segment grew, an upstream system started sending a field in a different format. The model is answering correctly about a world that no longer exists. Distinguished from a pipeline bug by its shape: it appears gradually, and it appears after a period of the model working.

Confirm

# Compare live input distributions against the training set, per feature
# (mean, median, null rate, and category frequencies for categoricals)

A feature whose distribution has moved materially since training — a null rate that climbed, a category that appeared or vanished, a mean that shifted

Fix

  • Retrain on recent data, and decide deliberately whether that is scheduled, triggered by a drift threshold, or manual — 'when somebody notices' is the default nobody chooses on purpose.
  • Monitor input distributions continuously. This needs no labels, which makes it a cheap place to start monitoring.
  • Check whether an upstream producer changed a format or a default; drift is sometimes somebody else's deploy.

Evaluation was optimistic because of leakage

Occasional

The production number is right and the evaluation number was wrong. Information that will not exist at prediction time leaked into training — a feature computed after the outcome, a random split that put near-duplicate rows on both sides, a target-derived aggregate. The gap was there from the first request, because there never was a better model.

Confirm

# Re-evaluate on a split by TIME rather than at random: train on earlier, test on later

Evaluation scores dropping sharply toward the production figure. If a time-based split reproduces the live number, the original evaluation was the problem

Fix

  • Split by time for anything where predictions are made about the future, which is most things.
  • Check every feature against the question: would this value be available, with this value, at the moment of prediction?
  • Be suspicious of a feature that improves the model a great deal on its own. Check it for leakage before you treat it as insight.

The serving stack runs different code or versions

Occasional

A tokeniser, an encoder or a scaler is a different version in the serving image than in the training environment. Library upgrades change defaults; a normalisation constant fitted during training is recomputed at serving instead of being loaded. The model artifact is identical and the thing feeding it is not.

Confirm

pip freeze > serving.txt   # in the serving image, then diff against the training environment

Version differences in anything that transforms data before the model sees it, and any preprocessing parameter recomputed at serving rather than loaded from the artifact

Fix

  • Ship preprocessing with the model as one versioned artifact rather than as code that happens to be deployed alongside it.
  • Pin the serving environment and rebuild training and serving images from the same base.
  • Persist fitted parameters — means, vocabularies, scaling constants — with the model and load them, never recompute.

The model is shaping the data it later learns from

Rare

The third cause Google names, and the least obvious. A recommender only ever collects feedback on what it chose to show, so the next training set is a record of its own preferences rather than of what users would have wanted. Performance can look stable on the data being collected while genuine quality narrows.

Confirm

# Compare outcomes on served items against a small randomised holdout that bypasses the model

The model looking strong on its own traffic and weak against randomly-served items — that gap is the loop

Fix

  • Keep a small randomised exploration slice so the training data is not entirely self-selected.
  • Record what was shown alongside what was chosen, so 'not clicked' can be told apart from 'never presented'.
  • Accept that this is a design problem rather than a bug to fix once.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

In plain terms

Your model scored well in testing but does worse on real traffic, and nothing crashes, so no alarm goes off. Either production feeds the model different data than testing did, the world has changed since training, or the test was easier than real life. Finding out which means comparing the two sides directly, so you need both of them recorded.

Words used on this page

Training-serving skew
Google's term for a difference between performance during training and performance during serving.
Feature
One input value the model uses, such as a count or a category, computed from raw data before the model sees it.
Feature vector
The full set of feature values for one prediction. Logging it at serving time is what makes skew diagnosable.
Distribution skew
When the distribution of feature values in training data is significantly different from serving data. Drift is the same comparison between consecutive periods.
Leakage
Information in training or evaluation data that will not exist at prediction time, such as test rows duplicated in training. It makes the evaluation look better than the model is.
Time-based split
Testing on data from after the training period, for example training on data up to 5 January and testing on 6 January onwards, as Google's rules recommend.
Feedback loop
The model's own outputs shaping the data it later learns from, for example collecting clicks only on items it chose to show.
Feature store
A shared system that supplies the same feature values to training and serving. Google's MLOps guide lists avoiding training-serving skew and reusing feature sets among its benefits.
What is actually happening

Google's Rules of Machine Learning calls this training-serving skew and defines it plainly: a difference between performance during training and performance during serving. The important property is that it is silent. No exception, no non-zero exit, no alert. The model returns confident answers that are worse than the ones it returned in evaluation.

The same document names three causes, and they are worth separating because the fixes have nothing in common: a discrepancy between how data is handled in the training and serving pipelines, a change in the data between training and serving, and a feedback loop between the model and the algorithm.

The first is an engineering bug. The second is drift and is expected — the world moves, and the answer is retraining rather than a fix. The third is subtle and specific: the model's own output changes the data it later learns from.

One honest caveat about this page: unlike the other entries here, there is no command that reproduces the symptom. Every confirmation below is a comparison between two things that should be identical and are not, which means you need both sides logged before you can diagnose it at all. That prerequisite is the single most useful thing to get right in advance.

How to tell this is your problem
WhereWhat you see
Live metrics vs the evaluation reportA gap that persists across days rather than a bad afternoon
The serviceEntirely healthy — no errors, normal latency, normal throughput
Who noticedUsually a business metric or a user complaint, not monitoring
TimingPresent from the first day of serving (a pipeline bug) or growing over weeks (drift)

How to know it is actually fixed

  • The offline and online feature vectors for the same input are identical, checked on a sample rather than on one example.
  • Live performance moves toward the evaluation figure after the fix, measured over days rather than hours.
  • The comparison that found it is now automated and runs on every training job, so the next instance fails a pipeline rather than a quarter.
  • If the fix was retraining, confirm the gap does not reopen at the same rate — if it does, the cadence is wrong rather than the model.

Stopping it happening again

  • Log the features actually used at serving time, and train from them where possible. This is the single highest-value habit in this list and it removes a whole class by construction.
  • Define each feature once, in one place, read by both paths.
  • Monitor input distributions from day one. It needs no labels, so there is no reason to wait.
  • Split by time when evaluating anything predictive, so the evaluation number means what you think it means.
  • Ship preprocessing and fitted parameters with the model as one artifact.
  • Decide the retraining policy deliberately and write it down.

Security considerations

  • Logging served feature vectors creates a new copy of production data, often personal data. Give it the same access control and retention as its source, and log a sample if that is enough: Google's rules say a small fraction is enough to check consistency.
  • Load preprocessing artifacts only from your own model registry. If they are Python pickles, loading one runs code, and Python's documentation says never to unpickle data that could have come from an untrusted source or been tampered with.
  • Training from logged serving data means whoever can send requests can influence the next model. Filter and validate logged inputs before they become training data, as you would any other untrusted input.
  • Run feature comparisons on production examples inside the production access boundary, or remove identifiers first. Do not paste raw records into notebooks, tickets or chat while chasing a mismatch.

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it

Where to go next