Lab: Correlating Logs Across a Multi-Service Failure
journalctl, Log Analysis, and Security Log Analysis each covered reading logs from one angle. A real production incident rarely announces itself in one log line from one service — it shows up as a cascade: an application error, followed by a systemd restart, followed by a spike in failed SSH attempts from an unrelated automated scanner that has nothing to do with the actual problem but adds noise while you are trying to diagnose it. This lab builds exactly that scenario and practices separating the signal from the noise using timestamps as the correlating thread.
Goal and End Result
By the end of this lab you will have reconstructed, from journalctl output alone, the correct sequence of events across two unrelated services during a simulated incident window, correctly distinguishing the actual root cause from a coincidental, unrelated log stream happening in the same time window.
Topology and Starting State
A single lab VM, using the greeter.service built in Service Management Lab if you completed it, or any custom service you can deliberately fail on demand.
Requirements
journalctl(part ofsystemd, present by default);- a second terminal session to generate simulated unrelated noise while diagnosing.
Security and Snapshot Note
This lab only reads logs and does not modify system state beyond starting/stopping a demo service — no rollback of system configuration is needed, only cleanup of the demo service itself if it was created for this lab.
Step 1: Establish a Baseline Time Window
Recording the exact wall-clock time before starting is the single most useful habit in log correlation — every journalctl --since/--until query in this lab anchors back to this moment, and without it, "narrow the search window" becomes guesswork.
Step 2: Generate a Real Failure
sudo systemctl stop greeter.service
sudo sed -i 's#/opt/greeter/greeter.sh#/opt/greeter/missing.sh#' /etc/systemd/system/greeter.service
sudo systemctl daemon-reload
sudo systemctl start greeter.service
Step 3: Generate Unrelated Noise at the Same Time
From a second terminal, simulate the kind of background noise that is present on almost any internet-facing server, regardless of whether anything is actually wrong:
This generates several genuine failed-login log entries in auth.log/the journal's sshd stream — coincidentally in the same time window as the real failure, but with no causal connection to it whatsoever. This is deliberately the hard part of the exercise: distinguishing correlation in time from causation in fact.
Step 4: Query Broadly First, Then Narrow
Following the same evidence-gathering discipline as Troubleshooting Workflow, start with a wide time window across everything, not one service in isolation:
Jul 29 10:31:05 host systemd[1]: greeter.service: Main process exited, code=exited, status=203/EXEC
Jul 29 10:31:05 host systemd[1]: greeter.service: Failed with result 'exit-code'.
Jul 29 10:31:20 host sshd[5210]: Failed password for invalid user nonexistent-user from 127.0.0.1 port 51422 ssh2
Jul 29 10:31:21 host sshd[5215]: Failed password for invalid user nonexistent-user from 127.0.0.1 port 51424 ssh2
At a glance, this output presents two unrelated events as though they might be connected — both showing up in the same -p err-filtered window is exactly what makes real incident logs confusing under time pressure.
Step 5: Isolate Each Service's Own Stream
Jul 29 10:31:05 host systemd[1]: Started greeter.service - Greeter demo service.
Jul 29 10:31:05 host systemd[1]: greeter.service: Failed to locate executable /opt/greeter/missing.sh: No such file or directory
Jul 29 10:31:05 host systemd[1]: greeter.service: Main process exited, code=exited, status=203/EXEC
Jul 29 10:31:05 host systemd[1]: greeter.service: Failed with result 'exit-code'.
Jul 29 10:31:20 host sshd[5210]: Failed password for invalid user nonexistent-user from 127.0.0.1 port 51422 ssh2
Filtering by -u <unit> — the exact technique from journalctl — separates the two independent stories cleanly: greeter.service's own log stream contains its complete, self-sufficient failure explanation, with no mention of SSH at all, confirming the two events share only a coincidental timestamp, not a cause.
Step 6: Build a Combined Timeline Only Once the Root Cause Is Confirmed
Only after Step 5 confirms the two events are independent does it make sense to present them on one timeline — for an incident report, not for diagnosis:
2026-07-29T10:31:05+00:00 host systemd[1]: greeter.service: Failed to locate executable /opt/greeter/missing.sh
2026-07-29T10:31:20+00:00 host sshd[5210]: Failed password for invalid user nonexistent-user from 127.0.0.1
-o short-iso produces unambiguous, sortable ISO-8601 timestamps — worth using by default in any incident report that might be read by someone in a different timezone, since the journal's default output format omits the year and timezone offset entirely.
Practical Scenario: Writing the Incident Note
Problem. A teammate, seeing both log lines above in a shared alert channel, asks whether the failed SSH logins caused the service to crash.
Check. Steps 5 confirmed each service's log stream independently — greeter.service's failure is fully explained by a missing executable, with zero reference to SSH, authentication, or any external actor.
Conclusion. The two events are unrelated; the SSH attempts are routine background noise (automated scanning is constant on any internet-reachable host — see Security Log Analysis for recognizing when this noise actually escalates into a real brute-force pattern worth acting on), and the correct incident note states the greeter.service root cause on its own, without implying a connection that the evidence does not support.
Troubleshooting Tips
journalctl --since/--untilreturns nothing, though you know events happened in that window. Confirm the system's timezone matches what you assumed withtimedatectl, and that you are not filtering with a unit name that has a typo —journalctlsilently returns empty output for a nonexistent unit rather than an error.- Too much unrelated noise to read through manually. Combine
-p err(priority filter) with-u <unit>(unit filter) together, rather than relying on either alone, to cut the search space down before reading line by line. - Assuming two log lines close in time are related. As this lab demonstrated directly, proximity in time is a hypothesis to test, not a conclusion — always isolate each service's own stream before asserting a causal link.
Rollback and Cleanup
sudo systemctl stop greeter.service
sudo sed -i 's#/opt/greeter/missing.sh#/opt/greeter/greeter.sh#' /etc/systemd/system/greeter.service
sudo systemctl daemon-reload
sudo systemctl start greeter.service
Final Assessment Criterion
This lab is complete when you can, from journalctl output alone:
- state the exact root cause of the
greeter.servicefailure, citing the specific log line; - confirm the SSH failed-login entries are unrelated by showing that
greeter.service's own filtered log stream fully explains its failure with no reference to authentication events; - produce a combined, correctly time-ordered summary of both independent events using
-o short-iso.
Exercises
- Repeat this lab, but this time cause the two failures to be genuinely related — for example, fill the disk enough to prevent a service from writing a required file (see Disk Usage Lab for a safe way to simulate this), and confirm that this time, isolating each service's stream reveals a shared underlying cause instead of two independent ones.
- Write a single
journalctlcommand that shows only prioritywarningand above, across every unit, restricted to the last 15 minutes, in ISO-8601 format. - Explain, in a short paragraph, why isolating each service's own log stream (Step 5) should always come before building a combined timeline (Step 6), rather than the other way around.