Skip to main content
DevOps Insightsguides & answers

systemd · troubleshooting

systemd service fails to start

systemd tried to start the service and it did not stay up, and the exit status tells you which of two very different things happened. Either systemd could not even launch your program — a wrong path, a missing user, a missing directory — or your program started and exited on its own. Those need opposite places to look.

First published

New to this? Read what is actually happening first, then come back to the steps.

On this page · 10 sections

Before you start

  • A shell on the machine, and sudo for anything that changes state: systemctl restart, systemctl reset-failed and systemctl daemon-reload.
  • Read access to the system journal: root, or membership of the systemd-journal group. The journalctl manual says adm and wheel also grant it, but those groups usually carry other privileges too.
  • The unit's exact name, such as myapp.service, and the account in its User= line, so you can run its ExecStart command as that user with sudo -u.
  • The systemd version, from systemctl --version. The manual text checked for this page is from systemd v262, and older releases can differ in detail.
  • Permission to edit the unit file or add a drop-in under /etc/systemd/system/, if the fix turns out to be in the unit.

Run this first

step 1 of 4
systemctl status <unit> --no-pager

Look for Active state and the main process's exit code and status

The exit status decides everything else. A systemd-defined code means look at the unit file; any other code means look at the program.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

What exit status does systemctl status show for the main process?

systemctl status <unit> --no-pager

Cause space

5 of 5 still possible

  • systemd could not execute the program (203)Common
  • The program started and exited on its ownCommon
  • It crashed repeatedly and hit the start rate limitCommon
  • The configured user could not be set up (217)Occasional
  • The working directory does not exist (200)Occasional

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Check it with a tool

Or diagnose it manually

In this order. The first command usually contains the whole answer.

  1. Step 1

    systemctl status <unit> --no-pager

    The exit status decides everything else. A systemd-defined code means look at the unit file; any other code means look at the program.

    Look for Active state and the main process's exit code and status

  2. Step 2

    journalctl -u <unit> -b --no-pager | tail -50

    What the program said before it exited, for this boot only. Output that holds only systemd's messages about the unit, with nothing from the program, is itself informative: the program may never have run.

    Look for An error naming a file, port, host, variable or permission

  3. Step 3

    systemctl cat <unit>

    Shows the unit file and its drop-ins as they are on disk. For the values systemd has actually loaded, use systemctl show <unit> -p ExecStart -p User -p WorkingDirectory. Those can differ from the file you edited until you run systemctl daemon-reload.

    Look for ExecStart, User, WorkingDirectory and Restart as they really are

  4. Step 4

    systemd-analyze verify "$(systemctl show -P FragmentPath <unit>)"

    Loads the unit file and reports errors in it without starting anything.

    Look for Warnings about the unit or units it references

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

systemd could not execute the program (203)

Common

ExecStart points at something that cannot be executed: the path is wrong, the file was moved by an upgrade, it lacks the execute permission, or a script's interpreter line names an interpreter that is not installed. systemd fails at the execve call itself, so nothing your program would log ever runs.

Confirm

systemctl cat <unit> | grep -E '^ExecStart' ; ls -l <path-from-ExecStart>

A path that does not exist, is not absolute, or is not marked executable

Fix

  • Correct ExecStart to the absolute path of an executable that exists.
  • For a script, make sure it is executable and that its first line names an interpreter that is installed at that path.
  • Reload unit definitions after editing, or systemd keeps using the old ExecStart.
sudo systemctl daemon-reload && sudo systemctl restart <unit> && systemctl status <unit> --no-pager

daemon-reload is what makes an edited unit file take effect.

The program started and exited on its own

Common

The exit status is an ordinary one from your program, so systemd launched it successfully and it chose to stop — a missing configuration file, a port already in use, a database it could not reach, an environment variable the unit never set. The reason is in what the program printed.

Confirm

journalctl -u <unit> -b --no-pager | tail -50

The program's last messages before it exited. An error naming a file, port, host or variable is the cause

Fix

  • Fix whatever the program reported. The unit file is not the problem in this case.
  • If it works when run by hand but not as a service, compare environments: a service does not inherit your shell's variables, working directory or PATH.
  • Run the exact ExecStart command as the service's user to reproduce it outside systemd.

It crashed repeatedly and hit the start rate limit

Common

With Restart= set, systemd restarts a crashing service — until it has been started more than StartLimitBurst times within StartLimitIntervalSec, after which the manual says it is not permitted to start any more. Within the rate-limit interval the service refuses to start even if the original problem is fixed. systemctl reset-failed flushes the counter at once; otherwise starts are allowed again when the interval has passed.

Confirm

systemctl show <unit> -p StartLimitBurst -p StartLimitIntervalUSec -p Restart ; journalctl -u <unit> -b --no-pager

The limits in force, and a journal showing the service starting and failing repeatedly in quick succession before it stopped

Fix

  • Find why it was crashing first — scroll the journal back past the restarts to the first failure, which is the real cause.
  • After fixing that, clear the failed state so systemd will start it again.
  • Do not disable the rate limit to make the symptom go away. It exists so a broken service cannot spin a CPU forever.
sudo systemctl reset-failed <unit> && sudo systemctl start <unit>

reset-failed clears the failed state and its counters. Without it, starts are refused until the StartLimitIntervalSec window has passed.

The configured user could not be set up (217)

Occasional

User= names an account that does not exist on this machine (a missing Group= or SupplementaryGroups= fails the same way, with status 216/GROUP) — common when a unit file is copied between hosts, or when a package that creates the account was not installed. systemd cannot switch to it, so the program never runs.

Confirm

systemctl cat <unit> | grep -E '^(User|Group)=' ; getent passwd <user>

A user or group that getent cannot find

Fix

  • Create the account, or point User= at one that exists.
  • If the unit was copied from another machine, check every account and path it names, not just this one.

The working directory does not exist (200)

Occasional

WorkingDirectory= names a directory that is missing or not accessible to the service's user — often a release directory removed by a deploy, or a mount that systemd does not manage and that was not up yet when the service started.

Confirm

systemctl cat <unit> | grep -E '^WorkingDirectory=' ; sudo -u <user> sh -c 'cd <dir>' && echo ok

A directory that is missing, or that the service user cannot enter

Fix

  • Create the directory, or correct the path.
  • If it lives on a mount, make sure the mount is a systemd mount unit. WorkingDirectory= already gives the service Requires= and After= on the mount units its path needs; a mount made outside systemd gets no such ordering.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

In plain terms

systemd is the service manager that starts your service and keeps track of it, and this error means the service did not stay running. The exit status it records tells you which of two things happened: your program never got to run, which points at the unit file, or it ran and quit, which points at the program's own output.

Reading systemctl status, from the manual

Documented output, not reproduced: systemd cannot run on the machine used for this page. This is the example in the systemctl(1) manual, as published in the systemd v262 sources. It shows a healthy unit; the notes below say which lines change when a unit fails, using the manual and the code that prints this screen.

Command

systemctl status bluetooth

Output · as shown in systemctl(1), status example (systemd v262 man page source)

● bluetooth.service - Bluetooth service
   Loaded: loaded (/usr/lib/systemd/system/bluetooth.service; enabled; preset: enabled)
   Active: active (running) since Wed 2017-01-04 13:54:04 EST; 1 weeks 0 days ago
     Docs: man:bluetoothd(8)
 Main PID: 930 (bluetoothd)
   Status: "Running"
    Tasks: 1
   Memory: 648.0K
      CPU: 435ms
   CGroup: /system.slice/bluetooth.service
           └─930 /usr/lib/bluetooth/bluetoothd

Jan 12 10:46:45 example.com bluetoothd[8900]: Not enough free handles to register service
Jan 12 10:46:45 example.com bluetoothd[8900]: Current Time Service could not be registered
Jan 12 10:46:45 example.com bluetoothd[8900]: gatt-time-server: Input/output error (5)

How to read it

  • The dot before the name summarises the state. The manual says it becomes a red cross (×) when the unit is failed, and a white circle (○) when it is inactive.
  • Active: is the line to read first. A service that did not stay up shows failed here; the manual says this state is entered after a crash, an error exit code or a timeout, and that the cause is logged.
  • Main PID: shows the running process. Once the process has exited, systemctl's code prints how it ended in brackets, as (code=exited, status=<number>/<name>). The name comes from systemd's table of codes, for example FAILURE for 1 or EXEC for 203.
  • Loaded: says whether the unit file itself could be used. The manual lists not-found when there is no unit file and bad-setting when an essential setting could not be parsed; either one means the problem is in the unit, not the program.
  • The lines at the bottom are the unit's most recent journal entries, 10 by default (-n changes the number). The manual points to journalctl -u bluetooth for the same kind of filter, and for earlier runs and earlier boots that status does not show.

Words used on this page

Unit
Anything systemd manages. A service unit is a file ending in .service that says which program to start and how.
ExecStart=
The unit setting that names the program to run. It takes an absolute path, or a bare name that systemd looks up in a fixed list of standard directories such as /usr/bin/.
Exit status
The number a process ends with; 0 means success. systemd defines its own codes from 200 upward, such as 203 EXEC, for failures that happen before your program runs.
failed state
Where a unit ends up after a crash, an error exit code or a timeout. Its exit code and status stay recorded until the unit is started again or reset with systemctl reset-failed.
Journal
systemd's log store. journalctl -u <unit> shows the unit's own messages plus systemd's messages about it, and -b limits that to the current boot.
Restart=
Whether systemd starts the service again after its process exits, is killed or times out. The default is no.
Start rate limit
A unit started more than StartLimitBurst= times within StartLimitIntervalSec= (5 times in 10 seconds by default) is refused further starts, including manual ones, until the interval has passed or systemctl reset-failed runs.
daemon-reload
systemctl daemon-reload makes systemd re-read every unit file. Until you run it, systemd keeps using the version it loaded before your edit.
What is actually happening

When a service fails, systemctl status shows it as failed along with the exit status of its main process. The useful question is not "why did it fail" but "did my program ever run", because the answer decides whether the explanation is in the unit file or in the program's own output.

systemd answers that for you. Its documentation explains that when the service manager fails to apply a unit's execution settings, the process exits with a non-zero code "before the configured command line is executed" — after systemd has created the process, but before it hands over to your binary. These failures use exit codes systemd defines itself: 200 EXIT_CHDIR when changing to the working directory failed, 203 EXIT_EXEC when executing the program failed, 217 EXIT_USER when the user could not be set up. Seeing one of those means your program never started, so its logs are empty for a reason.

203 is the one people meet most. The documentation describes it as the execve call failing and says it is "most likely" caused by "a missing or non-accessible executable file" — a typo in ExecStart, a bare program name that is not in systemd's fixed search path, where an absolute path is needed, a script without its execute bit, or a binary installed somewhere other than where the unit says.

Any other exit status is your program's own: it started, did something, and exited. systemd is only the messenger here, and the reason is whatever the program printed, which the journal captured.

There is a third state that looks like a failure to start and is really systemd protecting the machine. A unit configured to restart that keeps crashing will be restarted until it exceeds a rate limit, after which, in the words of the manual, it is "not permitted to start any more". From then on, starting it by hand is refused until StartLimitIntervalSec has passed (10 seconds by default) or the counter is flushed with systemctl reset-failed — so the visible symptom is a service that refuses to start, while the cause is the crash loop that came before.

How to tell this is your problem
WhereWhat you see
systemctl status <unit>Active: failed, with the main process's exit code and status
systemctl status <unit>An exit status of 203, 217 or 200 — systemd's own codes, meaning the program never ran
journalctl -u <unit> -bThe program's own last words before exiting. If it never started, you see only systemd's own messages about the unit and nothing from the program
systemctl start <unit>Returns an error immediately and the unit stays failed, after a run of restarts

How to know it is actually fixed

  • systemctl status shows the unit active (running), and it is still running a few minutes later rather than restarting.
  • journalctl -u <unit> -b shows a clean start with no repeated restart cycle.
  • The service survives a reboot, if it is enabled — a unit that only works when started by hand is not fixed.

Stopping it happening again

  • Use absolute paths in ExecStart and check them after every package or deploy that could move a binary.
  • Run the exact ExecStart command as the service's user before enabling the unit. Most failures reproduce immediately.
  • Validate unit files with systemd-analyze verify before reloading them.
  • Keep the start rate limit on. When a unit hits it, treat the first failure in the journal as the incident, not the refusal at the end.
  • Put configuration a service needs into the unit — Environment=, EnvironmentFile=, WorkingDirectory= — rather than relying on anything from an interactive shell.

Security considerations

  • Do not delete User= to get past status 217. For system services the default user is root, so the program would then run with full privileges. Create the missing account, or let systemd allocate one for each run with DynamicUser=.
  • When you fix status 203 with chmod +x or a new path, check that the program and every directory above it can be written only by root. Anyone who can replace that file runs their code as the service's user the next time it starts.
  • If the program failed for a missing setting, do not put a password or key into Environment=. The systemd.exec manual says unit environment variables are visible to unprivileged clients over D-Bus and are not suitable for secrets; it recommends LoadCredential= instead.
  • journalctl -u and systemctl status show whatever the program printed while it was failing, which can include connection strings or tokens. Read the output before you paste it into a ticket, and give helpers the systemd-journal group rather than adm or wheel.

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it

Where to go next