systemd Troubleshooting
Every tool covered so far — Service Management, Creating a Custom systemd Service, and journalctl — now comes together for one practical purpose: finding, systematically rather than by trial and error, why a service isn't behaving as expected. This article shows a diagnostic approach based on a definite sequence rather than random guessing — from finding failed units, to analyzing boot time, to identifying dependency problems.
What You Will Learn
By the end of this article you will be able to:
- see every failed unit on a system at once with
systemctl --failed; - read the exit-code and signal information in
systemctl statusand infer a likely cause; - analyze boot time with
systemd-analyze blameandcritical-chain; - inspect a unit's dependencies with
systemctl list-dependencies; - apply a consistent, step-by-step process for diagnosing a service failure.
Finding Failed Units
The fastest way to see how many services are having problems, and which ones:
UNIT LOAD ACTIVE SUB DESCRIPTION
● backend-demo.service loaded failed failed Backend demo application
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
1 loaded units listed.
This command is a shorthand for systemctl list-units --state=failed — much as is-failed from Service Management works for a single unit, --failed gives a system-wide overview. Whenever an unspecified problem shows up, this should be the very first command.
The Diagnostic Process, Step by Step
Once a failed unit is found, the following sequence usually clarifies the cause:
flowchart TD
A["systemctl --failed"] --> B["systemctl status <unit>"]
B --> C["journalctl -u <unit> -n 50"]
C --> D{"Is the cause clear?"}
D -->|No| E["systemctl list-dependencies <unit>"]
D -->|Yes| F["Fix and restart"]
E --> F
Step 1: An Overview with systemctl status
● backend-demo.service - Backend demo application
Loaded: loaded (/etc/systemd/system/backend-demo.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Fri 2026-07-24 12:01:15 UTC; 30s ago
Process: 7120 ExecStart=/usr/bin/python3 /opt/backend-demo/app.py (code=exited, status=1/FAILURE)
Main PID: 7120 (code=exited, status=1/FAILURE)
Two fields here carry the most information:
Result:— the general failure category from systemd's point of view (exit-code,signal,timeout, and others);code=/status=— the process's exact exit shape:exited(a normal exit, withstatus=giving the exit code) orkilled(terminated by a signal, withstatus=giving the signal name, e.g.9/KILL).
Result value |
Meaning |
|---|---|
exit-code |
The process exited on its own with a non-zero code (an error inside the program) |
signal |
The process was terminated by a signal (e.g. SIGKILL, SIGSEGV) |
timeout |
It failed to respond within TimeoutStartSec= or TimeoutStopSec= |
resources |
A resource allocation problem for the process (e.g. insufficient memory) |
When Result: exit-code comes from systemd failing to even launch the process — rather than from the program's own logic — the status= number is a specific systemd exec code that names the cause directly:
status= |
Name | What went wrong |
|---|---|---|
203/EXEC |
EXEC | The ExecStart= binary could not be run — wrong path, not executable, or a bad interpreter line |
200/CHDIR |
CHDIR | The WorkingDirectory= does not exist or isn't accessible |
217/USER |
USER | The User=/Group= account does not exist |
226/NAMESPACE |
NAMESPACE | A sandboxing directive failed to set up — e.g. a ReadWritePaths= path missing under ProtectSystem=strict |
These four resolve a large share of "my brand-new unit won't start" cases before reading a single application log line — the number points straight at the offending directive. A plain application error (a Python traceback, a bad config value), by contrast, exits with the program's own code (often 1/FAILURE) and is read in the next step.
A process shown as killed with status=9/KILL that nobody killed by hand is very often the OOM killer. Confirm it in the kernel log:
On recent systemd the status line itself may read Result: oom-kill, which removes the guesswork.
Step 2: Reading the Full Message with journalctl
systemctl status only shows the last few lines. For full context, use the filtering covered in journalctl:
More often than not, the real cause is hiding right here — in the application's own error message (a Python traceback, "Address already in use") — while systemctl status only shows the result from systemd's point of view.
Step 3: Checking Dependencies
If the unit itself looks fine but doesn't start when expected, or behaves in an unexpected order, the problem might be in a dependency:
This command shows, as a tree, the actual result of the Wants=/Requires= directives covered in systemd Units and Targets — a convenient way to visually check which unit depends on which.
Analyzing Boot Time: systemd-analyze
If the problem is specifically during boot, with some unit extending boot time more than it should, a dedicated tool is used.
Total boot time:
Seeing which unit took the most time:
blame lists units in descending order of how long each one took to start on its own — the unit at the top of the list is the leading candidate for slowing down the boot.
Viewing the longest sequential dependency chain:
graphical.target @9.912s
└─multi-user.target @9.910s
└─backend-demo.service @4.707s +5.203s
└─network.target @4.703s
└─network-online.target @4.701s
blame shows how long a unit took by itself; critical-chain shows the longest path of units that had to be waited for sequentially to reach a target — the two answer different questions, which is why it's worth using both together.
Info
A unit near the top of systemd-analyze blame is not always the actual culprit — for example, a unit waiting on network-online.target might appear to take a long time even though it does no heavy work itself. critical-chain makes that distinction clearer.
Reviewing All Errors at Once
This combines the -p (priority) filter from journalctl with the current boot, showing only err-level and more severe entries — useful for seeing every problem at once when several units are misbehaving simultaneously.
Common Mistakes
Treating systemctl status as Sufficient
status only gives a short summary. The actual cause (for example, the full error text from inside the application) is often only visible through journalctl -u <unit>.
Treating Result: exit-code and Result: signal as the Same Thing
exit-code means the program itself ended with an error (for example, a misconfiguration), while signal means it was terminated by an external force (kill, the OOM killer). Their causes and fixes are entirely different — the first calls for checking the code, the second for checking system resources or external influence.
Restarting a Unit Repeatedly Without Checking Its Dependencies
If the problem isn't the unit itself but another unit it depends on (for example, the network isn't ready yet), restarting just that one unit might help temporarily, but the underlying cause remains. Checking with systemctl list-dependencies breaks that cycle.
Exercises
- In a test VM, deliberately create a unit with a broken
ExecStart=path (pointing to a nonexistent file), rundaemon-reloadandstart, then find the cause usingsystemctl --failedandjournalctl -u. - Note the three slowest units from
systemd-analyze blameand estimate why each might be taking as long as it does. - Read
systemd-analyze critical-chainoutput and describe the longest sequential path tographical.target(ormulti-user.target). - Fix the deliberately broken unit's
ExecStart=path and confirm it disappears from thesystemctl --failedlist.
Verification criterion: given a failed unit's systemctl status output, you should be able to say whether the next step is reading the application's own log or checking its dependencies.
Summary
Diagnosing a systemd failure is not guesswork — it's a consistent process: first identify which unit is failing with systemctl --failed, then check the general result type (exit-code or signal) with systemctl status, then read the full error text with journalctl -u, and check dependencies with systemctl list-dependencies if needed. For boot-time problems specifically, systemd-analyze blame/critical-chain are dedicated, targeted tools. If the problem is severe enough that the system doesn't boot at all, the minimal recovery modes covered in the next article, Rescue and Emergency Mode, are needed.