Lab: Diagnosing "It Works by Hand but Not Under cron"
cron Basics and cron Environment and Logs explained why a script's environment under cron differs from an interactive shell's. This lab makes that abstract warning concrete by writing a script that genuinely works when run by hand, genuinely fails under cron, and walking through the exact diagnostic sequence that finds out why — the single most common cron complaint in real operations, and one every LFCS-style task involving scheduled jobs is built to test.
Goal and End Result
By the end of this lab you will have a script that fails silently under cron because of two independent script bugs — a PATH assumption and a relative-path assumption — that a missing output redirection initially hides from view entirely. You will make the failure visible first, then diagnose and fix each of the two bugs one at a time, and finally re-implement the same job as a systemd timer for comparison.
Topology and Starting State
A single lab VM or container with cron installed and running (sudo systemctl status cron).
Requirements
- a text editor;
- basic familiarity with crontab syntax from crontab and System cron.
Security and Snapshot Note
This lab only creates files under your own home directory and a crontab entry for your own user — no system-wide configuration is touched, and cleanup is a simple crontab -r plus removing the lab directory.
Step 1: Write a Script That Works Perfectly by Hand
mkdir -p ~/lab/cron-demo && cd ~/lab/cron-demo
cat <<'EOF' > report.sh
#!/bin/bash
which jq
cd data
cat today.txt | jq .
EOF
chmod +x report.sh
mkdir -p data
echo '{"status": "ok"}' > data/today.txt
Verify it works interactively:
If jq is not installed, install it first: sudo apt install jq — the script genuinely needs it, which is part of the point of this lab.
Step 2: Schedule It Under cron and Watch It Fail
Wait one minute, then check whether it actually did anything:
No new output appears anywhere, and there is no obvious error — this silence itself is the first and most important symptom: cron jobs fail silently by default unless output is explicitly captured, exactly the trap cron Environment and Logs warns about.
Before assuming the script failed, first confirm cron even attempted to run it — a job that produced no output because cron never fired (a crontab syntax error, the cron service stopped, or the wrong user's crontab) is a completely different problem from a job that ran and failed. The cron daemon logs every job it starts:
# Debian/Ubuntu: cron logs to syslog / the journal
journalctl -u cron --since "2 minutes ago"
grep CRON /var/log/syslog | tail -5
A CMD line naming your script confirms cron did fire it on schedule — which means the silence is coming from the script itself, not from a scheduling failure, and directs the rest of this lab toward the script's environment rather than the crontab entry. No CMD line at all would instead point at the crontab (check crontab -l and the cron service) before touching the script.
Step 3: Capture Output First — Before Diagnosing Anything Else
The single most useful first fix for any misbehaving cron job is making its failures visible at all:
Verify, after waiting for the next run:
/home/<username>/lab/cron-demo/report.sh: line 2: jq: command not found
/home/<username>/lab/cron-demo/report.sh: line 3: cd: data: No such file or directory
cat: today.txt: No such file or directory
Now the actual causes are visible, and there are two, independent of each other.
Step 4: Diagnose and Fix — Cause 1: PATH Doesn't Include jq's Location
Cron's default PATH is typically a bare minimum (/usr/bin:/bin), which usually does include /usr/bin — but many tools installed via a package manager into /usr/local/bin, a language version manager, or a custom install location are not on it, exactly as explained in cron Environment and Logs. Confirm where jq actually lives:
If this path is genuinely outside cron's default PATH on your system, add it explicitly at the top of the crontab, or reference the binary by its full path directly inside the script — the more robust of the two fixes, since it does not depend on crontab's PATH line being preserved if the crontab is ever recreated:
Step 5: Diagnose and Fix — Cause 2: The Working Directory Isn't What You Assumed
The cd data line assumed the script's working directory was ~/lab/cron-demo, which is true when run by hand from that directory, but cron does not start a job in the directory the crontab entry happens to reference — as cron Environment and Logs explains, cron jobs typically start in the user's home directory instead, regardless of where the script itself lives on disk.
Fix — use absolute paths inside the script, never a bare relative path:
cat <<'EOF' > ~/lab/cron-demo/report.sh
#!/bin/bash
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
/usr/bin/jq --version
cd "$SCRIPT_DIR/data"
cat today.txt | /usr/bin/jq .
EOF
chmod +x ~/lab/cron-demo/report.sh
SCRIPT_DIR, resolved from BASH_SOURCE[0] rather than assumed, makes the script's own location the anchor for every relative reference inside it — correct regardless of what directory the script happens to be invoked from, whether interactively, from cron, or from a systemd unit.
Verify:
Step 6: Reimplement the Same Job as a systemd Timer
# ~/.config/systemd/user/report.service
[Unit]
Description=Run report.sh
[Service]
Type=oneshot
ExecStart=/home/<username>/lab/cron-demo/report.sh
# ~/.config/systemd/user/report.timer
[Unit]
Description=Run report.sh every minute
[Timer]
OnCalendar=*-*-* *:*:00
Persistent=true
[Install]
WantedBy=timers.target
Verify:
Notice the diagnostic experience is immediately better: journalctl --user -u report.service shows the job's output and exit status directly, with no manual output redirection required — exactly the advantage systemd Timers highlights over plain cron for anything beyond a trivially simple job.
Common Mistakes
- Assuming a script's interactive success predicts its cron success. As this lab demonstrated directly, an interactive shell's
PATHand working directory are both richer than cron's minimal defaults — a script relying on either implicitly can pass every manual test and still fail under cron. - Diagnosing without first capturing output. Step 3's redirection should be the very first change made to any misbehaving cron job — diagnosing blind, based on silence alone, wastes time compared to reading the actual error text.
- Fixing the symptom (adding one missing binary to
PATH) without checking for a second, independent cause in the same script. This lab had two separate bugs; fixing only the first would have left the job still silently failing.
Rollback and Cleanup
crontab -r
systemctl --user disable --now report.timer
rm -f ~/.config/systemd/user/report.service ~/.config/systemd/user/report.timer
systemctl --user daemon-reload
rm -rf ~/lab/cron-demo
Final Assessment Criterion
This lab is complete when all of the following hold:
- you can state, from memory, the two specific reasons a script that runs perfectly by hand can fail identically under cron;
- the fixed script uses no relative paths and no bare command names that depend on
PATH; - the systemd timer equivalent runs successfully and its output is visible via
journalctl --user -u report.servicewith no manual redirection.
Exercises
- Remove the output redirection from the crontab entry again, deliberately reintroduce one of the two bugs, and time how long it takes you to diagnose it using only the process from this lab, without looking back at the fix.
- Add
MAILTO=""to the top of your crontab and explain, referencing cron Environment and Logs, what this line changes about how cron reports a job's output. - Convert a real, existing cron job on a lab machine (or one from an earlier project in this course, such as Backup Script) into a systemd timer, and compare the two side by side for ease of diagnosis after a deliberately introduced failure.