Skip to content

Lab: Building, Breaking, and Fixing a systemd Service

Custom systemd Service and systemd Troubleshooting each covered half of this lab's material — writing a unit, and diagnosing one that fails. This lab deliberately does both in sequence: build a working service, then break it three different, realistic ways, and diagnose and fix each one using only the standard systemctl/journalctl toolchain — the exact skill an LFCS-style "this service won't start, fix it" task is testing.

Goal and End Result

By the end of this lab you will have a custom systemd service (greeter.service) that starts correctly, survives a reboot, and — after being deliberately broken three separate times — is diagnosed and repaired each time using the standard systemd diagnostic sequence, with the reasoning behind each fix explained rather than guessed at.

Topology and Starting State

A single lab VM or container with systemd as PID 1 (true of any current Ubuntu Server LTS or RHEL-family installation, but not of most minimal Docker containers — this lab needs a real VM or a systemd-enabled container).

Requirements

Security and Snapshot Note

A custom unit file under /etc/systemd/system/ runs whatever ExecStart points to with the privileges of the User= it specifies — always set User= to an unprivileged account unless the service genuinely needs root, and never copy a unit file from an untrusted source without reading ExecStart first.

Step 1: Build the Working Service

sudo useradd -r -s /usr/sbin/nologin greeter-svc
sudo mkdir -p /opt/greeter
cat <<'EOF' | sudo tee /opt/greeter/greeter.sh
#!/bin/bash
while true; do
    echo "$(date '+%Y-%m-%d %H:%M:%S') - greeter service is alive"
    sleep 10
done
EOF
sudo chmod +x /opt/greeter/greeter.sh
sudo chown -R greeter-svc:greeter-svc /opt/greeter

# /etc/systemd/system/greeter.service
[Unit]
Description=Greeter demo service
After=network.target

[Service]
Type=simple
User=greeter-svc
ExecStart=/opt/greeter/greeter.sh
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now greeter.service

Verify:

systemctl status greeter.service
journalctl -u greeter.service -n 5

● greeter.service - Greeter demo service
     Loaded: loaded (/etc/systemd/system/greeter.service; enabled)
     Active: active (running)
...
Jul 29 10:00:10 host greeter.sh[4821]: 2026-07-29 10:00:10 - greeter service is alive

Step 2: Break It — Case 1: Wrong Executable Path

sudo sed -i 's#/opt/greeter/greeter.sh#/opt/greeter/greeter-typo.sh#' /etc/systemd/system/greeter.service
sudo systemctl daemon-reload
sudo systemctl restart greeter.service

Diagnose, following the process from systemd Troubleshooting:

systemctl status greeter.service

● greeter.service - Greeter demo service
     Active: failed (Result: exit-code)
   Main PID: 4903 (code=exited, status=203/EXEC)

203/EXEC specifically means systemd could not even execute the binary named in ExecStart — a strong, specific signal to check the path itself before assuming anything about the script's own logic.

The status=NNN/NAME codes systemd reports for a failure before your program's own code even runs are a small, fixed, worth-memorizing set — recognizing them tells you exactly which part of the unit is wrong without reading a single log line:

status= Meaning Where to look
203/EXEC The ExecStart binary could not be executed — wrong path, or not executable The ExecStart= path (Step 2) and its permission bit (Step 3)
200/CHDIR The WorkingDirectory= does not exist or is not accessible The WorkingDirectory= line
217/USER The User= account does not exist The User= line — the exact failure Exercise 1 induces
226/NAMESPACE A sandboxing directive (e.g. ProtectHome, ReadOnlyPaths) could not be set up The [Service] hardening directives
Any value ≥ 1 with no /NAME Your program ran and itself exited non-zero The application's own logs — this is a bug in the code, not the unit

The key distinction to state in an interview: a NNN/NAME code means systemd failed to set up or launch the process, so the fault is in the unit file; a bare numeric exit code with no /NAME means the process launched fine and then failed on its own, so the fault is in the program.

journalctl -u greeter.service -n 10 --no-pager

greeter.service: Failed to locate executable /opt/greeter/greeter-typo.sh: No such file or directory

Fix and verify:

sudo sed -i 's#/opt/greeter/greeter-typo.sh#/opt/greeter/greeter.sh#' /etc/systemd/system/greeter.service
sudo systemctl daemon-reload
sudo systemctl restart greeter.service
systemctl is-active greeter.service

active

Step 3: Break It — Case 2: Missing Execute Permission

sudo chmod -x /opt/greeter/greeter.sh
sudo systemctl restart greeter.service

Diagnose:

journalctl -u greeter.service -n 10 --no-pager

greeter.service: Failed to execute /opt/greeter/greeter.sh: Permission denied

This message, distinct from the 203/EXEC "not found" case in Step 2, points specifically at a permissions problem rather than a missing file — the exact reasoning skill Troubleshooting Workflow emphasizes: read the specific error text before forming a hypothesis, since "Permission denied" and "No such file or directory" point to entirely different fixes.

Fix and verify:

sudo chmod +x /opt/greeter/greeter.sh
sudo systemctl restart greeter.service
systemctl is-active greeter.service

Step 4: Break It — Case 3: A Unit File Typo That Prevents Loading Entirely

sudo sed -i 's/\[Service\]/[Service/' /etc/systemd/system/greeter.service
sudo systemctl daemon-reload

Diagnose:

sudo systemctl restart greeter.service

Failed to restart greeter.service: Unit greeter.service not found.

sudo systemctl status greeter.service

Unit greeter.service could not be found.

A malformed unit file can cause systemd to fail to parse it at all, in which case systemctl reports the unit as simply not existing — rather than "failed" — which is a confusing message the first time you see it, since the file is clearly present on disk.

sudo systemd-analyze verify greeter.service

Failed to load unit file /etc/systemd/system/greeter.service: [Service section starting at line 6 is not properly closed

systemd-analyze verify checks a unit file's syntax independently of trying to start it, and is the direct tool for exactly this failure mode — worth reaching for immediately whenever systemctl reports a unit as "not found" despite the file visibly existing on disk.

Fix and verify:

sudo sed -i 's/\[Service/[Service]/' /etc/systemd/system/greeter.service
sudo systemd-analyze verify greeter.service
sudo systemctl daemon-reload
sudo systemctl restart greeter.service
systemctl is-active greeter.service

Troubleshooting Tips

  • systemctl daemon-reload was skipped after editing the unit file. systemd caches unit definitions in memory; an edit to the file on disk has no effect until daemon-reload re-reads it — a genuinely common source of "I fixed it and it's still broken."
  • journalctl -u <service> shows nothing at all. Confirm the unit name is spelled exactly as it appears in systemctl list-units, including the .service suffix, and check journalctl --since isn't scoped to a window that excludes the failure.
  • The service restarts in a loop. Check Restart= and RestartSec= in the unit file — a service repeatedly crashing and restarting with no delay can flood the journal and make root-cause diagnosis harder; systemctl status shows a rising Main PID on each check if this is happening.

Rollback and Cleanup

sudo systemctl disable --now greeter.service
sudo rm -f /etc/systemd/system/greeter.service
sudo systemctl daemon-reload
sudo rm -rf /opt/greeter
sudo userdel greeter-svc

Final Assessment Criterion

This lab is complete when all of the following hold:

  • greeter.service starts, is enabled, and produces log lines visible via journalctl -u greeter.service;
  • for each of the three deliberately introduced failures, you can state — without re-reading this lab — the specific journalctl or systemd-analyze verify message that identified the cause;
  • you can explain the difference between a unit reported as failed (Steps 2 and 3) and a unit reported as not found (Step 4), and why systemd distinguishes the two.

Exercises

  1. Introduce a fourth failure of your own design — for example, setting User= to a nonexistent account — and diagnose it using the same process before checking whether your hypothesis was correct.
  2. Add Restart=on-failure and RestartSec=5, then reintroduce Step 2's broken path. Observe in journalctl -u greeter.service -f how systemd behaves as it repeatedly attempts and fails to restart the service.
  3. Use systemctl list-dependencies greeter.service to check what network.target ordering actually guarantees, and explain in a sentence why After=network.target does not guarantee the network is fully configured with a working IP address by the time this service starts.

Summary

Every failure in this lab was diagnosable through the same three tools, in the same order: systemctl status for a quick overview, journalctl -u <service> for the specific error message, and systemd-analyze verify for a unit file that will not even load. Log Analysis Lab continues directly from here, going deeper into reading journalctl output across multiple correlated services rather than one isolated unit.

References