Skip to main content
DevOps Insightsguides & answers

CI/CD · troubleshooting

Passes locally, fails in CI

The same commit behaves differently on two machines, so something about those machines differs. The list of candidates is short, and each one can be confirmed rather than guessed at.

Reproduced on npm 11.12.1, . Where the text says “reproduced”, the behaviour was run on that version, not taken from documentation.

First published

New to this? Read what is actually happening first, then come back to the steps.

On this page · 10 sections

Before you start

  • The failing job's full log and the commit SHA it ran, so you test exactly what CI tested.
  • The workflow file, to see the runner, the Node.js or other toolchain versions it sets up, the environment variables it defines and its cache steps.
  • The same toolchain versions on your machine as the workflow installs, so each comparison changes one thing at a time.
  • Permission to re-run jobs, and to delete caches if a cache turns out to be the cause.
  • For a pull request from a fork, the knowledge that GitHub passes it no secrets except a read-only GITHUB_TOKEN.

Run this first

step 1 of 4
git status --porcelain && git log --oneline -1

Look for A clean tree and the same SHA the pipeline ran

Confirms you are actually comparing the same commit. An uncommitted local change is the most embarrassing version of this and takes five seconds to rule out.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

Re-run the CI job with no code change. What happens?

This splits the whole problem in two before you investigate anything.

Cause space

7 of 7 still possible

  • Different dependency versions resolvedCommon
  • Tests depend on order, or on each otherCommon
  • An environment variable is missing or differentCommon
  • The filesystem is case-sensitive on the runner and not on your machineOccasional
  • A dependency is not up yet when the tests startCommon
  • A cache is carrying something from a previous runOccasional
  • The runner is smaller than your machineOccasional

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Check it with a tool

Or diagnose it manually

In this order. The first command usually contains the whole answer.

  1. Step 1

    git status --porcelain && git log --oneline -1

    Confirms you are actually comparing the same commit. An uncommitted local change is the most embarrassing version of this and takes five seconds to rule out.

    Look for A clean tree and the same SHA the pipeline ran

  2. Step 2

    rm -rf node_modules && npm ci   # or your ecosystem's reproducible install

    Rebuilds your dependencies the way the runner does. If the failure now reproduces locally, it was dependency drift and you can debug it on your own machine.

    Look for The CI failure appearing locally

  3. Step 3

    npx jest --randomize   # Jest; other frameworks have their own randomised-order flag

    Separates a coupled test from an environmental difference. These two causes look identical from the pipeline and need completely different fixes.

    Look for The failure reproducing once order changes

  4. Step 4

    git ls-files | grep -i '<filename from the error>'

    Catches the case-sensitivity class immediately, which is otherwise one of the most confusing because the file is visibly right there.

    Look for Stored case differing from the case used in the import

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

Different dependency versions resolved

Common

Your lockfile pins exact versions and the install command you use locally may not honour it. npm install can update the lockfile to satisfy a range; npm ci installs exactly what is locked and fails if the lockfile and manifest disagree. If CI uses one and you use the other, the two machines are running different code from the same commit.

Confirm

npm ci --dry-run 2>&1 | head -20

An error that the lockfile is out of sync with package.json, or a resolved version differing from what you have in node_modules

Fix

  • Use the reproducible install in both places — npm ci, yarn install --immutable (Yarn 2 and later, where it is on by default in CI; --frozen-lockfile is the Yarn 1 flag and only a deprecated alias now), pip install -r requirements.txt against pinned versions, or your ecosystem's equivalent.
  • Commit the lockfile, and treat a dirty lockfile after install as a failure rather than a diff to ignore.
  • Delete local node_modules and reinstall from the lockfile before concluding the runner is wrong.
rm -rf node_modules && npm ci

Reproduces the runner's install locally. It only deletes node_modules, which npm ci removes anyway, but the install runs your dependencies' install scripts. Add --ignore-scripts if you only need the tree.

Tests depend on order, or on each other

Common

A test that passes alone and fails in the suite is reading state another test left behind — a shared database row, a module-level singleton, a stubbed clock never restored. CI often runs tests in a different order or in parallel workers, which exposes the dependency your local sequential run happened to satisfy.

Confirm

npx jest --runInBand --randomize   # note the printed seed and replay a failing order with --seed=<n>; other frameworks have their own flag

The failure reproducing locally once order changes. If shuffling reproduces it, the test is coupled rather than the environment being different

Fix

  • Make each test set up and tear down its own state instead of relying on what ran before it.
  • Find the coupling by running the failing test alongside the suite and bisecting, not by re-running it alone — alone is the case that already passes.
  • Resist adding a sleep. It converts a deterministic failure into an intermittent one.

An environment variable is missing or different

Common

Your shell has variables accumulated from a .env file, a profile script or an earlier export. The runner has only what the workflow sets. A missing value often does not throw — it reads as undefined and the code takes a different branch, which is why the failure can appear far from the cause.

Confirm

env | cut -d= -f1 | sort > local-env-names.txt   # names only: values can be secrets   # then compare against the runner's printed environment

Variables present locally and absent in CI. Compare names only — never paste the values into a log or an issue

Fix

  • Declare every variable the application needs, and fail fast at startup when one is missing rather than defaulting silently.
  • Keep a committed .env.example listing the names with placeholder values so the required set is documented.
  • Set them in the workflow or the CI provider's secret store, not in the image.

The filesystem is case-sensitive on the runner and not on your machine

Occasional

macOS and Windows default to case-insensitive filesystems; Linux runners are case-sensitive. import './Utils' finds utils.ts locally and finds nothing in CI. Git also preserves the case it first recorded, so a later rename that differs only in case may not have been committed at all.

Confirm

git ls-files | grep -i '<the filename from the error>'

The name Git actually stores, compared against the case used in the import that fails

Fix

  • Correct the import to match the stored filename exactly.
  • If the file itself is misnamed, rename it through Git so the change is recorded: git mv Utils.ts utils.ts.
  • git mv records a case-only rename directly (reproduced with Git 2.50.1 on macOS APFS). If an older Git refuses, rename through an intermediate name.

A dependency is not up yet when the tests start

Common

Locally your database has been running for days. In CI it starts seconds before the tests, and a container being started is not the same as a database being ready to accept connections. The failure looks like a connection error in the first test and nowhere else, which makes it read as a flaky test.

Confirm

<your CI client> logs <service-container> | head -30

The service's own readiness line, and whether its timestamp is before or after the first test's connection attempt

Fix

  • Wait for readiness rather than for the container to exist — a health check, or a short retry loop against the real protocol.
  • Where the CI provider supports service health checks, use them; they gate the job rather than being raced by it.
  • Prefer a bounded retry to a fixed sleep. A sleep is either too short on a slow day or wasted on every fast one.

A cache is carrying something from a previous run

Occasional

Dependency and build caches are keyed on a hash, and a key that does not include everything the cache depends on will restore stale content. This is the one case where the runner has something your clean machine does not, and it typically appears right after a dependency or toolchain change.

Confirm

# Re-run the job with caching disabled, or change the cache key, then compare

The failure disappearing without a code change, which implicates the cache rather than the commit

Fix

  • Include the lockfile hash and the toolchain version in the cache key so a change misses the exact key. A restore-keys prefix still restores the most recent older cache on a miss, so check the cache-hit output before trusting what was restored.
  • Never cache the build output itself unless you can prove the key covers every input.
  • Clear the cache when a dependency upgrade behaves inexplicably; it is cheap and it rules a whole class out.

The runner is smaller than your machine

Occasional

Hosted runners have modest CPU and memory. A test suite that fits comfortably in 32GB locally can be killed at 8 GB (the standard Linux runner for private repositories; 16 GB on public ones), and a parallel build that saturates two cores runs slower than the timeout allows. The symptom is usually a kill or a timeout rather than an assertion failure.

Confirm

# In the failing job, print limits before the step:  ulimit -a; nproc; free -m

An exit code of 137 — see the exit-code explainer — or a step ending exactly on the timeout boundary

Fix

  • Reduce parallelism in CI rather than assuming the local setting transfers.
  • Split a long suite into jobs that run concurrently instead of one job doing everything.
  • Raise the runner size only after confirming the workload genuinely needs it.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

In plain terms

Your tests or build pass on your computer and fail on the CI server for the same commit. That means the two machines differ in some way: installed package versions, settings, file names, timing or leftovers from earlier runs. Find the difference and you have found the bug.

A lockfile out of sync

Reproduced with npm 11.12.1 (Node.js 25.9.0) on a scratch project with two local file: dependencies, dep and dep2, run offline so nothing was downloaded. package.json listed both, and the committed package-lock.json listed only dep, as happens when someone adds a dependency and commits package.json without the updated lockfile.

Command

npm ci --dry-run 2>&1 | head -7

Output

npm error code EUSAGE
npm error
npm error `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm error
npm error Missing: dep2@1.0.0 from lock file
npm error Missing: dep2@1.0.0 from lock file
npm error

How to read it

  • npm ci refuses because package.json and package-lock.json disagree. npm's documentation says it exits with an error here instead of updating the lockfile, which is the step a plain npm install on your machine would have quietly taken.
  • Missing: dep2@1.0.0 from lock file names the dependency that package.json asks for and the lockfile lacks. It printed twice in this run; it is one problem.
  • Run without the pipe, the command exited with status 1, so the same mismatch fails a CI step on its own.
  • With the lockfile back in sync and a stray package left in node_modules, the same command printed its plan instead: add dep2 1.0.0, add dep 1.0.0 and remove stray 9.9.9, and left node_modules untouched. A remove line is something your machine has that a clean install would not.

Words used on this page

Runner
The machine that runs a CI job. Each GitHub-hosted runner is a new virtual machine; the standard Linux one has 2 CPUs and 8 GB of RAM for private repositories and 4 CPUs and 16 GB for public ones.
Lockfile
A file such as package-lock.json that records the exact version of every dependency, so two installs produce the same tree.
npm ci
npm's clean install. It installs exactly what the lockfile says, removes any existing node_modules first, and exits with an error instead of updating the lockfile when it disagrees with package.json.
Case-insensitive filesystem
One where Utils.js and utils.js are the same file, such as APFS on macOS or NTFS on Windows as usually set up. Linux runners treat them as two different names.
core.ignoreCase
A Git setting that git init and git clone turn on for case-insensitive filesystems. With it on, Git keeps the name it first recorded when only the letter case changes on disk.
Service container
A database or other service a CI job starts next to the tests. A health check such as --health-cmd pg_isready makes the job wait until it is ready.
Cache key and restore keys
The string a CI cache is saved under. An exact match is a cache hit; on a miss, restore-keys restore the most recent cache that matches a prefix, which can be older than you expect.
Test order seed
The number Jest uses to shuffle tests when you pass --randomize. Jest prints it, and --seed=<n> replays the same order.
What is actually happening

Nothing mysterious is happening. Your machine and the runner disagree about something — a dependency version, an environment variable, the filesystem, the clock, how many things run at once, or what was left over from last time. The disagreement is the bug; the failing test is just where it surfaced.

The single most useful reframing: your laptop is a long-lived, mutated environment and a GitHub-hosted runner is a fresh virtual machine. A self-hosted runner is the exception: it need not be clean for each job, so it can carry leftovers too. Most of these causes are things your machine acquired over months and the runner never had. That is why the runner is usually right and the laptop is usually the liar.

The reverse also happens and is rarer: the runner has something your machine does not, usually a cache from a previous run. Both directions are covered below.

How to tell this is your problem
WhereWhat you see
The pipelineA test or build step failing that succeeds when you run the same command locally
Re-running the jobSometimes green, sometimes red, with no code change — which points hard at ordering or concurrency
The diffNothing obviously related to the failure, because the cause is environmental rather than in the change

How to know it is actually fixed

  • The pipeline passes on a re-run with no code change.
  • The same command run locally after a clean reproducible install also passes.
  • Run the suite twice more. A cause you fixed by chance and a cause you fixed on purpose look the same after one green run.
  • If the fix was a test-ordering one, confirm it with the shuffle seed that originally failed rather than with the default order.

Stopping it happening again

  • Use the reproducible install command in both places, and fail the build if the lockfile changes during install.
  • Run the test suite in a randomised order locally too, so coupling is found by the person who wrote it.
  • Fail fast on missing configuration at startup instead of defaulting to a value that changes behaviour quietly.
  • Develop in the same base image CI uses where practical — it eliminates the filesystem, locale and toolchain classes at once.
  • Treat an intermittent test as a bug with an owner rather than a thing to re-run. Re-running is how a suite stops meaning anything.

Security considerations

  • Compare variable names, never values. GitHub redacts registered secrets in logs, but anything else, including a value derived from a secret, prints as-is unless you mask it with ::add-mask::.
  • If a job fails only on pull requests from forks because a secret is empty, that is by design. Do not move the workflow to pull_request_target to make it pass; GitHub warns that running untrusted code there can lead to cache poisoning and unintended access to secrets and write privileges.
  • Keep credentials out of cached paths, such as an .npmrc that holds a registry token. GitHub's caching reference warns that anyone with read access can open a pull request and read a cache's contents.
  • Reproducing the runner's install runs your dependencies' install scripts on your machine. Start with npm ci --dry-run, which changed nothing in the reproduction, or add --ignore-scripts when you only need the dependency tree.

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it

Where to go next