Skip to main content
DevOps Insightsguides & answers

CI/CD · troubleshooting

Passes locally, fails in CI

The same commit behaves differently on two machines, so something about those machines differs. The list of candidates is short, and each one can be confirmed rather than guessed at.

Run this first

step 1 of 4
git status --porcelain && git log --oneline -1

Look for A clean tree and the same SHA the pipeline ran

Confirms you are actually comparing the same commit. An uncommitted local change is the most embarrassing version of this and takes five seconds to rule out.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

Re-run the CI job with no code change. What happens?

This splits the whole problem in two before you investigate anything.

Cause space

7 of 7 still possible

  • Different dependency versions resolvedCommon
  • Tests depend on order, or on each otherCommon
  • An environment variable is missing or differentCommon
  • The filesystem is case-sensitive on the runner and not on your machineOccasional
  • A dependency is not up yet when the tests startCommon
  • A cache is carrying something from a previous runOccasional
  • The runner is smaller than your machineOccasional

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Check it with a tool

Or diagnose it manually

In this order. The first command usually contains the whole answer.

  1. Step 1

    git status --porcelain && git log --oneline -1

    Confirms you are actually comparing the same commit. An uncommitted local change is the most embarrassing version of this and takes five seconds to rule out.

    Look for A clean tree and the same SHA the pipeline ran

  2. Step 2

    rm -rf node_modules && npm ci   # or your ecosystem's reproducible install

    Rebuilds your dependencies the way the runner does. If the failure now reproduces locally, it was dependency drift and you can debug it on your own machine.

    Look for The CI failure appearing locally

  3. Step 3

    <your runner> --shuffle   # or the equivalent randomised-order flag

    Separates a coupled test from an environmental difference. These two causes look identical from the pipeline and need completely different fixes.

    Look for The failure reproducing once order changes

  4. Step 4

    git ls-files | grep -i '<filename from the error>'

    Catches the case-sensitivity class immediately, which is otherwise one of the most confusing because the file is visibly right there.

    Look for Stored case differing from the case used in the import

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

Different dependency versions resolved

Common

Your lockfile pins exact versions and the install command you use locally may not honour it. `npm install` can update the lockfile to satisfy a range; `npm ci` installs exactly what is locked and fails if the lockfile and manifest disagree. If CI uses one and you use the other, the two machines are running different code from the same commit.

Confirm

npm ci --dry-run 2>&1 | tail -20

An error that the lockfile is out of sync with package.json, or a resolved version differing from what you have in node_modules

Fix

  • Use the reproducible install in both places — `npm ci`, `yarn install --frozen-lockfile`, `pip install -r requirements.txt` against pinned versions, or your ecosystem's equivalent.
  • Commit the lockfile, and treat a dirty lockfile after install as a failure rather than a diff to ignore.
  • Delete local node_modules and reinstall from the lockfile before concluding the runner is wrong.
rm -rf node_modules && npm ci

Reproduces the runner's install locally. Safe: it only removes downloaded dependencies.

Tests depend on order, or on each other

Common

A test that passes alone and fails in the suite is reading state another test left behind — a shared database row, a module-level singleton, a stubbed clock never restored. CI often runs tests in a different order or in parallel workers, which exposes the dependency your local sequential run happened to satisfy.

Confirm

<your runner> --runInBand --shuffle   # jest; use the equivalent seed/shuffle flag for your framework

The failure reproducing locally once order changes. If shuffling reproduces it, the test is coupled rather than the environment being different

Fix

  • Make each test set up and tear down its own state instead of relying on what ran before it.
  • Find the coupling by running the failing test alongside the suite and bisecting, not by re-running it alone — alone is the case that already passes.
  • Resist adding a sleep. It converts a deterministic failure into an intermittent one.

An environment variable is missing or different

Common

Your shell has variables accumulated from a .env file, a profile script or an earlier export. The runner has only what the workflow sets. A missing value often does not throw — it reads as undefined and the code takes a different branch, which is why the failure can appear far from the cause.

Confirm

env | sort > /tmp/local.env   # then compare against the runner's printed environment

Variables present locally and absent in CI. Compare names only — never paste the values into a log or an issue

Fix

  • Declare every variable the application needs, and fail fast at startup when one is missing rather than defaulting silently.
  • Keep a committed .env.example listing the names with placeholder values so the required set is documented.
  • Set them in the workflow or the CI provider's secret store, not in the image.

The filesystem is case-sensitive on the runner and not on your machine

Occasional

macOS and Windows default to case-insensitive filesystems; Linux runners are case-sensitive. `import './Utils'` finds `utils.ts` locally and finds nothing in CI. Git also preserves the case it first recorded, so a later rename that differs only in case may not have been committed at all.

Confirm

git ls-files | grep -i '<the filename from the error>'

The name Git actually stores, compared against the case used in the import that fails

Fix

  • Correct the import to match the stored filename exactly.
  • If the file itself is misnamed, rename it through Git so the change is recorded: `git mv Utils.ts utils.ts`.
  • A case-only rename sometimes needs an intermediate name on a case-insensitive filesystem.

A dependency is not up yet when the tests start

Common

Locally your database has been running for days. In CI it starts seconds before the tests, and a container being *started* is not the same as a database being *ready to accept connections*. The failure looks like a connection error in the first test and nowhere else, which makes it read as a flaky test.

Confirm

<your CI client> logs <service-container> | head -30

The service's own readiness line, and whether its timestamp is before or after the first test's connection attempt

Fix

  • Wait for readiness rather than for the container to exist — a health check, or a short retry loop against the real protocol.
  • Where the CI provider supports service health checks, use them; they gate the job rather than being raced by it.
  • Prefer a bounded retry to a fixed sleep. A sleep is either too short on a slow day or wasted on every fast one.

A cache is carrying something from a previous run

Occasional

Dependency and build caches are keyed on a hash, and a key that does not include everything the cache depends on will restore stale content. This is the one case where the runner has something your clean machine does not, and it typically appears right after a dependency or toolchain change.

Confirm

# Re-run the job with caching disabled, or change the cache key, then compare

The failure disappearing without a code change, which implicates the cache rather than the commit

Fix

  • Include the lockfile hash and the toolchain version in the cache key so a change invalidates it.
  • Never cache the build output itself unless you can prove the key covers every input.
  • Clear the cache when a dependency upgrade behaves inexplicably; it is cheap and it rules a whole class out.

The runner is smaller than your machine

Occasional

Hosted runners have modest CPU and memory. A test suite that fits comfortably in 32GB locally can be killed at 7GB, and a parallel build that saturates two cores runs slower than the timeout allows. The symptom is usually a kill or a timeout rather than an assertion failure.

Confirm

# In the failing job, print limits before the step:  ulimit -a; nproc; free -m

An exit code of 137 — see the exit-code explainer — or a step ending exactly on the timeout boundary

Fix

  • Reduce parallelism in CI rather than assuming the local setting transfers.
  • Split a long suite into jobs that run concurrently instead of one job doing everything.
  • Raise the runner size only after confirming the workload genuinely needs it.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

What is actually happening

Nothing mysterious is happening. Your machine and the runner disagree about something — a dependency version, an environment variable, the filesystem, the clock, how many things run at once, or what was left over from last time. The disagreement is the bug; the failing test is just where it surfaced.

The single most useful reframing: your laptop is a long-lived, mutated environment and the runner is a fresh one. Most of these causes are things your machine acquired over months and the runner never had. That is why the runner is usually right and the laptop is usually the liar.

The reverse also happens and is rarer: the runner has something your machine does not, usually a cache from a previous run. Both directions are covered below.

How to tell this is your problem
WhereWhat you see
The pipelineA test or build step failing that succeeds when you run the same command locally
Re-running the jobSometimes green, sometimes red, with no code change — which points hard at ordering or concurrency
The diffNothing obviously related to the failure, because the cause is environmental rather than in the change

How to know it is actually fixed

  • The pipeline passes on a re-run with no code change.
  • The same command run locally after a clean reproducible install also passes.
  • Run the suite twice more. A cause you fixed by chance and a cause you fixed on purpose look the same after one green run.
  • If the fix was a test-ordering one, confirm it with the shuffle seed that originally failed rather than with the default order.

Stopping it happening again

  • Use the reproducible install command in both places, and fail the build if the lockfile changes during install.
  • Run the test suite in a randomised order locally too, so coupling is found by the person who wrote it.
  • Fail fast on missing configuration at startup instead of defaulting to a value that changes behaviour quietly.
  • Develop in the same base image CI uses where practical — it eliminates the filesystem, locale and toolchain classes at once.
  • Treat an intermittent test as a bug with an owner rather than a thing to re-run. Re-running is how a suite stops meaning anything.

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Related

Not authorized to perform sts:AssumeRoleWithWebIdentityContainer exits immediatelyCI/CD