Monitoring, Logging, and Automation at Fleet Scale

Every log and metrics tool covered earlier in this course — journalctl, dmesg, /proc, top — was framed around one server at a time: SSH in, run the command, read the output. That workflow stops scaling somewhere between ten and a hundred servers, not because the individual tools are wrong, but because a human checking each host by hand cannot keep pace with a fleet, and cannot be awake at 3 a.m. when a disk fills up. This article covers what changes when the same fundamentals — logs, metrics, and scheduled tasks — need to work across many hosts at once, unattended: centralizing what was local, and turning scheduled checks into automated responses.

What You Will Learn

By the end of this article you will be able to:

explain why per-host log files and manual checks fail to scale past a small number of servers;
forward journald logs to a central destination;
explain the Prometheus pull model and read a node_exporter metrics endpoint directly;
write an alerting rule that fires on a condition, not just a log line;
extend cron/systemd-timer automation from a single host to a fleet using the same Ansible model from the previous article.

Why Centralization Is Not Optional at Scale

A single server's logs, read with journalctl, answer "what happened on this machine." A fleet-wide incident — a bad deployment rolled out to fifty instances, a shared upstream dependency failing for every service that calls it — needs a different question answered: "what happened across every machine, correlated by time." Reconstructing that by SSHing into each host in turn during an active incident is exactly the failure mode centralized logging exists to prevent.

journalctl --since "10 minutes ago" -p err

This command, covered in journalctl, is the right first step on one host. At fleet scale, the equivalent question — "which hosts logged an error in the last ten minutes" — requires every host's logs to already be flowing somewhere queryable as a single dataset.

Forwarding journald to a Central Destination

systemd-journal-remote and systemd-journal-upload ship with systemd itself and forward journal entries over HTTP(S) to a central journal server without introducing a third-party dependency:

sudo apt install systemd-journal-remote

# /etc/systemd/journal-upload.conf
[Upload]
URL=https://log-collector.internal:19532

sudo systemctl enable --now systemd-journal-upload

For teams already standardized on a broader observability stack, rsyslog forwarding to a central syslog server, or shipping logs into Loki (Grafana's log aggregation system, designed to pair with Prometheus) or an ELK/OpenSearch stack, are the more common production choices — the specific destination matters less than the underlying principle: every host's logs need to leave that host and land in one place a human or an alerting rule can query without first deciding which of fifty machines to SSH into.

Interview angle: "why not just grep every server's logs with a for loop over SSH?"

This works for a quick, one-off check across a handful of hosts, and is a legitimate technique worth knowing (for h in $(cat hosts.txt); do ssh "$h" journalctl --since -10m -p err; done). It breaks down for three reasons: it does not scale past dozens of hosts within a usable timeframe, it provides no historical record once a host's local log rotates the entry away, and it depends on every host being reachable over SSH at the exact moment of the incident — precisely when an overloaded, half-failing host is least likely to respond quickly. Centralized logging decouples "can I query this event" from "is that specific machine currently reachable and responsive."

Metrics: The Prometheus Pull Model

Where logs record discrete events, metrics record numeric measurements over time — CPU usage, memory pressure, request latency, disk free space — sampled at a regular interval, which is what makes graphing a trend and alerting on a threshold possible in the first place. Prometheus, the dominant open-source metrics system in current infrastructure, uses a pull model: instead of each host pushing its metrics somewhere, Prometheus itself periodically fetches (scrapes) a plain HTTP endpoint each monitored host exposes.

sudo apt install prometheus-node-exporter
sudo systemctl enable --now prometheus-node-exporter
curl -s http://localhost:9100/metrics | head -n 15

# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 837291.42
node_cpu_seconds_total{cpu="0",mode="user"} 4213.11
# HELP node_memory_MemAvailable_bytes Memory available for starting new applications.
# TYPE node_memory_MemAvailable_bytes gauge
node_memory_MemAvailable_bytes 6.442450944e+09

node_exporter is deliberately simple: it reads the same kernel interfaces already covered in The /proc Filesystem — CPU time, memory, disk stats — and republishes them as plain-text HTTP, in a format Prometheus's server can scrape on a schedule (commonly every 15–60 seconds) and store as a time series.

The # TYPE line above each metric is not decoration — it names the metric type, and the two most common types behave differently enough that confusing them is a classic interview stumble:

A counter (node_cpu_seconds_total, marked counter) only ever increases, resetting to zero only when the process restarts. Its raw value is meaningless on its own — nobody cares that a CPU has spent 837,291 cumulative seconds idle. What matters is its rate of change, which is why counters are almost always wrapped in rate(): rate(node_cpu_seconds_total{mode="idle"}[5m]) gives idle CPU-seconds per second over the last five minutes, i.e. current idle fraction.
A gauge (node_memory_MemAvailable_bytes, marked gauge) can go up and down and represents a value that is meaningful right now — available memory, disk free space, temperature. A gauge is read directly, without rate(), because its instantaneous value is the answer.

The rule of thumb: if the question is "how fast is this happening," it is a counter and you need rate(); if the question is "what is the current level," it is a gauge and you read it as-is. Applying rate() to a gauge, or reading a counter's raw total as if it meant something, are two of the most common mistakes in a first Prometheus query.

# prometheus.yml (on the Prometheus server, not the monitored host)
scrape_configs:
  - job_name: "node"
    static_configs:
      - targets:
          - "web-01.internal:9100"
          - "web-02.internal:9100"

The pull model has a direct, practical consequence: if Prometheus cannot reach a host's /metrics endpoint at all, that absence is itself informative — a host that stops responding to scrapes is either down, network-partitioned, or the exporter crashed, and Prometheus surfaces this as the target being down rather than simply having no data. A push-based system, by contrast, has to distinguish "nothing to report" from "the host is gone" by some other means, since silence looks the same either way.

Alerting: Turning a Threshold Into a Page

A metric with no alert attached is a graph someone has to remember to look at — alerting closes that gap by evaluating a condition continuously and notifying a human (or triggering automation) the moment it becomes true.

# alert_rules.yml
groups:
  - name: node-alerts
    rules:
      - alert: HighDiskUsage
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) < 0.10
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Disk usage above 90% on {{ $labels.instance }}"

expr is the condition being evaluated against live metrics — here, available space on / dropping below 10% of total capacity. for: 10m requires the condition to hold continuously for ten minutes before the alert actually fires, which exists specifically to avoid paging someone for a brief, self-correcting spike (a log rotation temporarily filling disk before compression frees the space back up, for instance) — tuning this window correctly is a real, recurring operational judgment call: too short and alerts become noise nobody trusts; too long and a genuinely urgent problem sits unactioned.

There is a division of labour here that interviews probe directly: Prometheus only evaluates the rule and decides the alert is firing — it does not send the notification. When an alert fires, Prometheus forwards it to a separate component, Alertmanager, whose entire job is turning firing alerts into actual notifications: routing them to the right destination by label (severity: warning to a Slack channel, severity: critical to PagerDuty), grouping related alerts so a fifty-host outage produces one notification rather than fifty, deduplicating the same alert arriving from redundant Prometheus servers, and silencing alerts during planned maintenance. This split is why "write an alert rule" and "make sure the right person gets paged" are two distinct configuration tasks — the rule lives in Prometheus, the paging logic lives in Alertmanager.

This is the direct, scaled-up successor to the manual disk check in Disk Full Troubleshooting — the same df-derived signal, now evaluated continuously across every host in the fleet instead of only when someone happens to run the command by hand.

Interview angle: "how do you avoid alert fatigue?"

Alert fatigue — so many low-value alerts firing that real ones get ignored — is one of the most common operational failures in monitoring setups, and interviewers ask about it because it reveals whether a candidate has run on-call in practice. The concrete answer: alert on symptoms that require action (a service actually failing to respond, disk genuinely about to fill), not on every metric crossing an arbitrary threshold; use for: windows to suppress transient noise; and route alerts by actual severity so a warning does not page someone at 3 a.m. the same way a critical does. A monitoring setup with a hundred silent-by-default dashboards and one well-tuned alert is more useful in practice than one with fifty different alerts firing weekly.

Automation Across a Fleet: Extending cron and systemd Timers

Cron Basics and systemd Timers covered scheduling a recurring task on one host. At fleet scale, the scheduling mechanism on each individual host does not need to change — a systemd timer is still the right tool for "run this job on this machine every night" — but deploying and keeping that timer unit consistent across every host does, which is exactly the problem Infrastructure as Code Foundations covered Ansible solving for configuration in general:

# ansible playbook excerpt: ship the same systemd timer to every web host
- name: Deploy log cleanup timer
  hosts: web
  become: true
  tasks:
    - name: Install the cleanup script
      copy:
        src: files/log-cleanup.sh
        dest: /usr/local/bin/log-cleanup.sh
        mode: "0755"

    - name: Install the systemd service unit
      copy:
        src: files/log-cleanup.service
        dest: /etc/systemd/system/log-cleanup.service

    - name: Install the systemd timer unit
      copy:
        src: files/log-cleanup.timer
        dest: /etc/systemd/system/log-cleanup.timer

    - name: Enable and start the timer
      systemd:
        name: log-cleanup.timer
        enabled: true
        state: started
        daemon_reload: true

Running this playbook against the web inventory group installs and enables the identical log cleanup script and timer on every host in that group, idempotently — a second run makes no changes if nothing drifted, exactly as covered in Infrastructure as Code Foundations. This is the practical answer to "how do a hundred servers all run the same nightly job reliably": not a hundred people remembering to set up a hundred cron jobs by hand, but one playbook applied to a fleet defined in one inventory file.

Practical Scenario: Diagnosing a Fleet-Wide Disk Alert

Problem. The HighDiskUsage alert defined above fires for three of twelve web servers simultaneously.

Check. Because metrics are already centralized, the first step is a single Prometheus query across the whole fleet, not three separate SSH sessions:

node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}

This immediately shows whether the three affected hosts share something in common — the same deployment, the same log-rotation configuration — visible only because all twelve hosts' data sits in one queryable system.

Command. On one of the three affected hosts, confirm the specific cause the way Disk Full Troubleshooting already covers:

sudo du -xh --max-depth=2 / | sort -rh | head -n 10

12G   /var/log
9.8G  /var/log/nginx

Result. All three affected hosts received the same deployment two days ago, which enabled a debug-level access log that the log cleanup script's rotation policy was not configured to account for.

Conclusion. The alert correctly identified a real, fleet-wide problem the moment it crossed the threshold on the first host; centralized metrics turned "three separate incidents" into "one root cause affecting three hosts," found in a single query instead of three independent investigations.

Common Mistakes

Treating a dashboard as a substitute for an alert. A dashboard only helps if someone is actively looking at it; the disk-full scenario above would have been caught hours earlier by an alert, regardless of whether anyone happened to be watching a graph.
Setting alert thresholds without a for: window. A metric that briefly crosses a threshold and self-corrects (a short-lived CPU spike during a scheduled job) generates a false alarm without one, training responders to distrust the alerting system.
Deploying a scheduled job to each host by hand "just this once." This is the fleet-scale version of the imperative-script problem from Infrastructure as Code Foundations — it works until the fleet changes and someone forgets which hosts actually got the manual change.
Assuming a scrape target being down in Prometheus means the metric itself is fine, just "temporarily missing." As covered above, an unreachable target is itself a signal — it usually means the host, the exporter, or the network path between them has failed, not merely that there is no data to show.

Exercises

Install prometheus-node-exporter on a lab VM, query its /metrics endpoint with curl, and identify which line reports available memory and which reports root filesystem usage.
Write a Prometheus alert rule (it does not need a running Prometheus server to write correctly) for CPU usage sustained above 90% for five minutes, and explain what value you would choose for for: and why.
Using the Ansible model from Infrastructure as Code Foundations, write a playbook that deploys a systemd timer of your choice to a group of lab hosts, and confirm a second run of the playbook reports no changes.
In a short paragraph, explain why the pull model (Prometheus scraping hosts) makes an unreachable host visibly down, while a push model (hosts sending metrics somewhere) can make the same failure look identical to "the host simply had nothing new to report."

Summary

Fleet-scale operations replace per-host manual checks with three centralized mechanisms: log forwarding so events across every host are queryable as one dataset, a Prometheus-style pull-based metrics system so a host's silence is itself a detectable signal, and alerting rules that turn a threshold crossing into a notification instead of a graph nobody is watching. The scheduling and configuration-management tools from earlier articles — systemd timers, Ansible — do not change at fleet scale; what changes is that they get deployed and kept consistent across every host through the same infrastructure-as-code discipline, rather than repeated by hand. DevOps: Next Steps closes this module by connecting everything covered here to where a working Linux administrator typically goes from here.