Skip to main content
DevOps Insightsguides & answers

MLOps · troubleshooting

The model performs worse in production than in evaluation

Evaluation said one thing and production says another. Nothing errors, nothing restarts, and the service is healthy — which is why this is usually found late and by somebody outside engineering.

Run this first

step 1 of 5
# Was the gap there on day one, or did it grow?

Look for The shape of the metric since deployment, not its current value

This single question separates an engineering bug from drift, and they share no fixes. A gap present from the first request is a pipeline or evaluation problem; one that grew is the world moving.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

Was the gap present from the first day of serving, or did it grow over time?

This is the highest-value question available and it splits the causes almost cleanly.

Cause space

6 of 6 still possible

  • A feature is computed differently in training and servingCommon
  • The features used at serving time were never loggedCommon
  • The input data has changed since trainingCommon
  • Evaluation was optimistic because of leakageOccasional
  • The serving stack runs different code or versionsOccasional
  • The model is shaping the data it later learns fromRare

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Or diagnose it manually

In this order. The first command usually contains the whole answer.

  1. Step 1

    # Was the gap there on day one, or did it grow?

    This single question separates an engineering bug from drift, and they share no fixes. A gap present from the first request is a pipeline or evaluation problem; one that grew is the world moving.

    Look for The shape of the metric since deployment, not its current value

  2. Step 2

    # Score one identical production example through the training path and the serving path

    The most direct test available. If the feature vectors differ, you have found it and nothing else needs investigating.

    Look for Any feature that differs between the two

  3. Step 3

    # Compare live input distributions against the training set, per feature

    Needs no labels and no ground truth, so it works immediately and on any model. Catches drift and several format changes.

    Look for A null rate, category set or mean that has moved since training

  4. Step 4

    # Re-evaluate with a time-based split instead of a random one

    Rules out the possibility that production is fine and the original evaluation was optimistic.

    Look for Offline scores falling toward the production number

  5. Step 5

    pip freeze  # in the serving image, diffed against the training environment

    Cheap, and catches the class where the model is identical and its preprocessing is not.

    Look for Different versions of anything that transforms input, or parameters recomputed at serving

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

A feature is computed differently in training and serving

Common

The same named feature is produced by two pieces of code — an offline batch job and an online request path — and they do not agree. Different rounding, a different time window, refunds included in one and not the other, a null handled as zero in one and dropped in the other. The model was trained on one definition and is asked about the other on every single request.

Confirm

# Score the SAME production example through both paths and diff the feature vectors
# (no CLI for this — it is a comparison you have to arrange)

Any feature whose value differs between the offline and online computation for one identical input. One mismatched feature is enough to explain a large gap

Fix

  • Compute the feature once, in one place, and have both training and serving read that. This is the entire reason feature stores exist, and the benefit comes from the single definition rather than from buying a product.
  • Where you cannot unify immediately, add an automated comparison that fails the pipeline when the two implementations disagree on a sample of real inputs.
  • Treat a feature definition as an interface with one owner, not as code that happens to appear twice.

The features used at serving time were never logged

Common

Not a cause of the gap so much as the reason you cannot find it. If only the model's inputs at training time are recorded, there is nothing to compare against. Google's guidance is direct about this: save the set of features used at serving time, and pipe those to a log to use at training time.

Confirm

# Check whether your serving path records the feature vector it scored,
# not merely the request and the prediction

Absence of the served feature vector. If it is missing, this is the first thing to fix — every other diagnosis below depends on it

Fix

  • Log the exact feature vector that was scored, with the model version and a request identifier.
  • Sample rather than logging everything if volume is a concern; a consistent sample is enough for distribution comparison.
  • Then train from those logged features where you can, which removes this class of skew by construction.

The input data has changed since training

Common

The pipeline is correct and the world moved. Traffic shifted to mobile, a customer segment grew, a upstream system started sending a field in a different format. The model is answering correctly about a world that no longer exists. Distinguished from a pipeline bug by its shape: it appears gradually, and it appears after a period of the model working.

Confirm

# Compare live input distributions against the training set, per feature
# (mean, median, null rate, and category frequencies for categoricals)

A feature whose distribution has moved materially since training — a null rate that climbed, a category that appeared or vanished, a mean that shifted

Fix

  • Retrain on recent data, and decide deliberately whether that is scheduled, triggered by a drift threshold, or manual — 'when somebody notices' is the default nobody chooses on purpose.
  • Monitor input distributions continuously. This needs no labels, which makes it the cheapest useful monitoring in ML and the first thing to add.
  • Check whether an upstream producer changed a format or a default; drift is sometimes somebody else's deploy.

Evaluation was optimistic because of leakage

Occasional

The production number is right and the evaluation number was wrong. Information that will not exist at prediction time leaked into training — a feature computed after the outcome, a random split that put near-duplicate rows on both sides, a target-derived aggregate. The gap was there from the first request, because there never was a better model.

Confirm

# Re-evaluate on a split by TIME rather than at random: train on earlier, test on later

Evaluation scores dropping sharply toward the production figure. If a time-based split reproduces the live number, the original evaluation was the problem

Fix

  • Split by time for anything where predictions are made about the future, which is most things.
  • Check every feature against the question: would this value be available, with this value, at the moment of prediction?
  • Be suspicious of a feature that improves the model a great deal on its own. That is more often leakage than insight.

The serving stack runs different code or versions

Occasional

A tokeniser, an encoder or a scaler is a different version in the serving image than in the training environment. Library upgrades change defaults; a normalisation constant fitted during training is recomputed at serving instead of being loaded. The model artifact is identical and the thing feeding it is not.

Confirm

pip freeze > serving.txt   # in the serving image, then diff against the training environment

Version differences in anything that transforms data before the model sees it, and any preprocessing parameter recomputed at serving rather than loaded from the artifact

Fix

  • Ship preprocessing with the model as one versioned artifact rather than as code that happens to be deployed alongside it.
  • Pin the serving environment and rebuild training and serving images from the same base.
  • Persist fitted parameters — means, vocabularies, scaling constants — with the model and load them, never recompute.

The model is shaping the data it later learns from

Rare

The third cause Google names, and the least obvious. A recommender only ever collects feedback on what it chose to show, so the next training set is a record of its own preferences rather than of what users would have wanted. Performance can look stable on the data being collected while genuine quality narrows.

Confirm

# Compare outcomes on served items against a small randomised holdout that bypasses the model

The model looking strong on its own traffic and weak against randomly-served items — that gap is the loop

Fix

  • Keep a small randomised exploration slice so the training data is not entirely self-selected.
  • Record what was shown alongside what was chosen, so 'not clicked' can be told apart from 'never presented'.
  • Accept that this is a design problem rather than a bug to fix once.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

What is actually happening

Google's Rules of Machine Learning calls this training-serving skew and defines it plainly: a difference between performance during training and performance during serving. The important property is that it is silent. No exception, no non-zero exit, no alert. The model returns confident answers that are worse than the ones it returned in evaluation.

The same document names three causes, and they are worth separating because the fixes have nothing in common: a discrepancy between how data is handled in the training and serving pipelines, a change in the data between training and serving, and a feedback loop between the model and the algorithm.

The first is an engineering bug and the most common. The second is drift and is expected — the world moves, and the answer is retraining rather than a fix. The third is subtle and specific: the model's own output changes the data it later learns from.

One honest caveat about this page: unlike the other entries here, there is no command that reproduces the symptom. Every confirmation below is a comparison between two things that should be identical and are not, which means you need both sides logged before you can diagnose it at all. That prerequisite is the single most useful thing to get right in advance.

How to tell this is your problem
WhereWhat you see
Live metrics vs the evaluation reportA gap that persists across days rather than a bad afternoon
The serviceEntirely healthy — no errors, normal latency, normal throughput
Who noticedUsually a business metric or a user complaint, not monitoring
TimingPresent from the first day of serving (a pipeline bug) or growing over weeks (drift)

How to know it is actually fixed

  • The offline and online feature vectors for the same input are identical, checked on a sample rather than on one example.
  • Live performance moves toward the evaluation figure after the fix, measured over days rather than hours.
  • The comparison that found it is now automated and runs on every training job, so the next instance fails a pipeline rather than a quarter.
  • If the fix was retraining, confirm the gap does not reopen at the same rate — if it does, the cadence is wrong rather than the model.

Stopping it happening again

  • Log the features actually used at serving time, and train from them where possible. This is the single highest-value habit in this list and it removes a whole class by construction.
  • Define each feature once, in one place, read by both paths.
  • Monitor input distributions from day one. It needs no labels, so there is no reason to wait.
  • Split by time when evaluating anything predictive, so the evaluation number means what you think it means.
  • Ship preprocessing and fitted parameters with the model as one artifact.
  • Decide the retraining policy deliberately and write it down.

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Related

Passes locally, fails in CICI/CD