Lab: Mixed Troubleshooting Scenarios
Every lab in this module isolated one domain — permissions, users, services, logs, networking, SSH, cron, disk space. A real incident, and a real LFCS-style exam task, rarely announces which domain it belongs to; the symptom is simply "the application is broken," and the first job is figuring out where to even start looking. This closing lab presents five independent, self-contained scenarios in that same undifferentiated form, each solved using the five-step troubleshooting process — define the problem precisely, gather evidence, form a hypothesis, test one change, document — as the organizing method rather than domain-specific intuition alone.
What You Will Practice
- correctly identifying which domain a vague symptom actually belongs to, before reaching for any specific tool;
- applying the same five-step process regardless of domain;
- resisting the urge to change multiple things at once when the first hypothesis is wrong;
- reading each scenario's evidence completely before forming a hypothesis, rather than pattern-matching to a superficially similar problem seen before.
How to Use This Lab
For each scenario, the symptom and initial evidence are given exactly as a user or monitoring system would report them. Before reading the diagnosis section, write down your own hypothesis and the single next command you would run to test it. This matters: the value of this lab is in practicing the reasoning under uncertainty, not in reading a solution — treat each scenario as if you were the only person able to fix it.
Scenario 1: "The Website Returns a 502 Error Intermittently"
Symptom. Users report the site sometimes loads normally, sometimes shows "502 Bad Gateway." It is not consistent — reloading the page often fixes it.
Initial evidence:
Diagnosis. An intermittent 502 from a reverse proxy (nginx, in front of an application server) means the proxy itself is up — it is actively serving a response, just an error one — while the backend it forwards to is not reliably answering. This is the same distinction between the proxy layer and the application layer failing independently that any reverse-proxy setup needs to account for.
connect() failed (111: Connection refused) while connecting to upstream, upstream: "http://127.0.0.1:3000/"
The service reports running, yet nginx sometimes cannot connect to it — check whether the application is actually crashing and being restarted repeatedly, which systemctl status alone (showing only current state) can miss:
Fourteen restarts in ten minutes confirms the application is crash-looping — nginx's 502 is a downstream symptom of an upstream service that is only intermittently up at the moment a request happens to arrive.
Root cause and fix. Reading the application's own crash log (journalctl -u myapp.service -n 50, the next natural step) is where this scenario would continue — the lesson here stops at correctly redirecting attention from the proxy layer to the application layer, which is the step most commonly skipped under pressure, since the 502 error visibly comes from nginx.
Scenario 2: "I Can SSH In, but sudo Suddenly Stopped Working for Everyone"
Symptom. Every user on a shared server reports sudo now fails with a permissions-related error, immediately after a colleague says they "cleaned up some old sudoers rules this morning."
Initial evidence:
Diagnosis. This message is unusually specific and self-explanatory once read carefully — sudo refuses to honor any file under /etc/sudoers.d/ unless it is owned by root, as a defense against a non-root user being able to modify their own privilege rules. The colleague's "cleanup" almost certainly involved editing the file with a regular text editor as a non-root user, or copying it in a way that changed its ownership, rather than using visudo -f.
Fix. Since sudo itself is unusable by anyone right now, this requires the console/root-fallback access referenced throughout SSH Configuration Lab — from a root session:
Verify:
Lesson. This scenario exists specifically to reinforce sudo Security's warning about visudo-only editing — and to demonstrate that when sudo itself is broken for everyone, the fix requires a privilege path independent of sudo, which is exactly why maintaining root console access, as emphasized in SSH Configuration Lab, is not optional.
Scenario 3: "Backups Have Been Silently Failing for a Week"
Symptom. A scheduled backup job appears in crontab -l, no one has received any error notification, but the backup destination directory has no files newer than eight days.
Initial evidence:
Diagnosis. No error notification does not mean no error occurred — as cron Environment and Logs covers, a cron job's output only reaches someone if MAILTO is configured correctly and mail delivery on the host actually works, which is a surprisingly common silent gap: mail delivery infrastructure quietly broken (no local MTA configured) will drop cron's output with no indication anywhere that anything went wrong.
No output redirection at all — following the exact diagnostic instinct from cron Automation Lab, the first fix is making failures visible, then finding the actual cause:
Root cause. /data was likely a mount point that stopped being mounted (an fstab entry removed, a network share disconnected) eight days ago — matching the timeline exactly. Confirm:
Fix. Remount the source (or fix the underlying /etc/fstab entry, per fstab), then add output capture to the crontab entry so this class of failure is never silent again:
Lesson. "No error reported" and "no error occurred" are not the same statement — a monitoring gap (missing output capture, broken mail delivery) can hide a real failure for an arbitrarily long time, and the fix for the immediate problem (remount /data) is incomplete without also fixing the fix's own visibility.
Scenario 4: "A New Server Can Reach the Internet but Not Our Internal Database"
Symptom. A freshly provisioned application server can curl external sites successfully but times out connecting to the internal database server on the same private network.
Initial evidence:
Diagnosis. Internet reachability confirms basic routing and DNS resolution work generally — the failure is specific to one internal destination. Following Network Diagnostics' layered approach, check name resolution and reachability separately before assuming the database service itself is at fault:
Name resolution succeeds — the address is a legitimate internal one, not a typo or stale DNS entry.
The host itself is reachable at the network layer — this rules out a routing problem and narrows the hypothesis specifically to the database's listening port.
Root cause. This is a Case 2-style symptom from Network Troubleshooting Lab — the host is reachable, but the specific port is not, which points at either the database not listening on that interface, or a firewall rule on the database host itself blocking this new server's IP specifically (most likely, since other application servers presumably connect successfully). Check the database host's firewall:
The new application server's IP (10.0.4.20, say) was never added to this allow list when it was provisioned.
Fix and verify:
Lesson. A new server provisioned from an image or IaC template inherits the template's software but not necessarily an existing peer's individually granted network exceptions — this is exactly the kind of gap Infrastructure as Code Foundations discusses: firewall rules added by hand, one server at a time, are precisely the drift that declarative, version-controlled infrastructure is meant to eliminate.
Scenario 5: "The Log Analysis Script Reports Different Results Each Run"
Symptom. A script meant to count failed SSH login attempts from the past hour reports a different, seemingly random count every time it runs, even when run twice in immediate succession with (apparently) no new login attempts in between.
Initial evidence:
Diagnosis. A script whose output changes without any change in the underlying data usually means the script's time window itself is shifting between runs, not that the data is actually different — following Log Analysis Lab's emphasis on anchoring time windows precisely:
Root cause. --since "1 hour ago" is evaluated fresh, relative to the exact moment the script runs, every single time — two runs a few minutes apart have genuinely different one-hour windows, each potentially including or excluding different failed-login events near the boundary. This is not a bug in journalctl; it is the script author assuming "the last hour" is a fixed, comparable window when it is actually a constantly sliding one.
Fix — anchor the window explicitly if reproducibility matters:
#!/bin/bash
WINDOW_START="$(date -d '1 hour ago' '+%Y-%m-%d %H:%M:%S')"
journalctl -u ssh --since "$WINDOW_START" | grep -c "Failed password"
Even this fix only makes the window explicit and loggable — it does not make two runs at different times return the same count, because the underlying event stream genuinely differs between them; the fix's real value is that the script's own logged WINDOW_START now explains why two runs differ, rather than leaving that as a mystery.
Lesson. A script producing inconsistent results is not necessarily broken — it may be doing exactly what it was told, against a moving target its author did not realize was moving. Confirming what a command's inputs actually are (here, the literal time window --since resolves to) before suspecting the tool itself is a repeatedly useful diagnostic instinct across every domain in this course.
Exercises: Solve These Without a Walkthrough
The following scenarios are given as symptom and initial evidence only — apply the five-step process yourself, and verify your diagnosis is correct before considering the exercise complete rather than checking against a provided answer.
- "The server ran out of memory and killed our application, but
free -hshows plenty of memory available right now." Initial evidence:dmesg | grep -i "out of memory"shows an OOM-killer entry from twenty minutes ago naming your application's process. Diagnose why the current, healthy-lookingfree -houtput does not contradict a real OOM event that already happened and was already resolved by the kernel killing the offending process. Verification: you should be able to state, usingjournalctlanddmesgtimestamps together, the exact process and approximate memory usage that triggered the kill. - "A
systemdtimer job that has run reliably for months suddenly stopped running, with no configuration changes." Initial evidence:systemctl --user list-timersshows the timer as active, butjournalctl --user -u <service>shows no new runs in three days. Diagnose the most likely cause connected specifically to user (not system) timers and session lifecycle. Verification: your diagnosis should explain why a system-level (root) timer would not have this same failure mode. - "Two administrators both insist their own
ssh_configHostalias is correct, but only one of them can connect." Initial evidence: both configs defineHost db-serverwith differentHostNamevalues. Diagnose which configuration file actually takes effect and why, referencing SSH Config's coverage of configuration precedence. Verification: you should be able to point to the specific line inssh -v's output that reveals which config file and whichHostblock was actually applied.
Summary
Every scenario in this lab was solvable with the same five steps regardless of which domain it turned out to belong to: a precise problem statement, evidence gathered before a hypothesis, one hypothesis tested with one change, and a documented result. The specific tools differed — journalctl, ufw, ls -l, nc, dmesg — but the reasoning discipline connecting them did not, and that discipline, not memorized command syntax, is what an LFCS-style task and a real production incident both actually test.