Process Monitoring: top, htop, and Reading Load in Real Time

ps from Programs and Processes gives you a single frozen frame. Most real diagnostic work — "why is this server slow right now," "is one process hogging the CPU," "is memory pressure building up" — needs a moving picture instead: continuously refreshed process state, updated system load, and the ability to sort, filter, and act on what you see without leaving the screen. That is what top and htop provide.

This article covers reading top's header correctly (a section most people skim past, which is exactly where load average, memory pressure, and CPU breakdown live), interpreting htop's process list and colored meters, and using both tools to actually diagnose — not just observe — a load problem.

top: The Header Before the Process List

Run top with no arguments and the first five or six lines, before the process table, contain more diagnostic information than the sortable list below them:

top - 14:32:07 up 6 days,  3:12,  2 users,  load average: 4.15, 2.03, 0.98
Tasks: 187 total,   2 running, 184 sleeping,   0 stopped,   1 zombie
%Cpu(s): 62.3 us,  8.1 sy,  0.0 ni, 28.5 id,  0.9 wa,  0.0 hi,  0.2 si,  0.0 st
MiB Mem :  15928.4 total,    412.6 free,   9820.1 used,   5695.7 buff/cache
MiB Swap:   2048.0 total,   1890.2 used,    157.8 free.   4103.9 avail Mem

Load Average

load average: 4.15, 2.03, 0.98 reports the average number of processes in a runnable or uninterruptible-sleep state over the last 1, 5, and 15 minutes, respectively. It is not a percentage and it is not bounded at 100 — its meaningful ceiling depends on how many CPU cores the machine has.

The rule of thumb: divide the load average by the CPU core count. A load of 4.15 on a 4-core machine (4.15 / 4 ≈ 1.04) means the system is, on average, very slightly over-subscribed — a bit more runnable work than cores available at any instant. The same 4.15 on a 16-core machine (4.15 / 16 ≈ 0.26) indicates the system is comfortably under-loaded.

nproc

Comparing the three numbers against each other tells you about a trend: 4.15, 2.03, 0.98 (1-minute higher than 5-minute, higher than 15-minute) means load is currently rising; the reverse ordering means it is settling down after a spike.

The CPU Breakdown Line

%Cpu(s): 62.3 us, 8.1 sy, 0.0 ni, 28.5 id, 0.9 wa, 0.0 hi, 0.2 si, 0.0 st breaks total CPU time into eight categories:

A high sy relative to us often points to excessive system call overhead (heavy context switching, small I/O operations that should be batched, or a misbehaving application making far more syscalls than necessary). On a cloud VM, persistent nonzero st (steal time) means the physical host is oversubscribed and your instance is not getting its fair share of CPU even though your own workload has not changed — a signal to check the hosting provider's status or consider a different instance size, not something to try to fix inside the guest.

Tasks and Memory Lines

Tasks: 187 total, 2 running, 184 sleeping, 0 stopped, 1 zombie is a live count matching the STAT codes from ps. A small, stable number of zombies is usually harmless (a parent simply hasn't called wait() yet); a zombie count that keeps growing indicates a parent process that never reaps its children — a real bug, covered further in Signals and Process Control.

The memory line's used/free/buff/cache split is explained in depth in Process Memory; the short version is that free alone dramatically understates truly available memory, because the kernel deliberately uses spare RAM for disk cache, which it will hand back instantly if an application needs it.

Interacting with top While It Runs

top is not just a static display — several single-key commands change what and how it shows, without restarting it:

Running top -H shows individual threads instead of one line per process — essential when a single multi-threaded process (a java server, postgres with parallel workers) is consuming CPU and you need to know which specific thread inside it is responsible.

top -H -p "$(pgrep -d, -x nginx)"

-p restricts top to a specific, comma-separated list of PIDs; pgrep -d, -x nginx supplies that list, matching the exact process name and joining results with commas.

htop: A More Readable Alternative

htop is not preinstalled on a minimal Ubuntu Server image; install it if it is missing:

sudo apt update
sudo apt install htop

htop's value over plain top is mostly ergonomic rather than functional: colored per-core meters at the top, mouse support, scrollable and horizontally scrollable process lists (useful for long command lines), a built-in tree view (F5 or the t key), and a visible function-key menu at the bottom so you do not need to memorize commands.

Key differences worth knowing when moving between the two:

• Meters: the top bars show each CPU core individually by default, plus separate memory and swap bars, color-coded by category (green for user time, red for kernel time, blue for low-priority niced time).
• Tree view (F5): shows the same parent-child relationship as pstree, but live and interactive — useful for identifying which specific worker process under a multi-process server (nginx, postgres) is the one consuming resources.
• F9 (Kill): prompts for a signal by name, not just SIGTERM — convenient for sending SIGHUP to reload a configuration without fully restarting a process, a pattern covered in Signals and Process Control.
• F4 (Filter) and F3 (Search): narrow the visible list by a text match against the command line, which is faster than scrolling through hundreds of processes for a specific service.

A Realistic Scenario: Diagnosing a Sudden Load Spike

Problem. An alert fires: load average has climbed sharply on a web server, and response times are degrading. You need to determine, within a couple of minutes, whether this is CPU contention, I/O contention, or a runaway single process, before deciding whether to restart anything.

Check first, without changing anything:

uptime

14:41:03 up 12 days,  6:02,  1 user,  load average: 11.82, 6.44, 2.10

The rising trend (11.82 > 6.44 > 2.10) confirms the spike is recent and ongoing, not a stale reading.

Open top and read the header first, before looking at any individual process:

top

%Cpu(s): 18.2 us,  4.1 sy,  0.0 ni,  9.5 id, 67.9 wa,  0.0 hi,  0.3 si,  0.0 st

wa at 67.9% with us/sy both low is the decisive clue: the CPUs are mostly idle, waiting on I/O, not saturated with computation. This immediately rules out "a process is burning CPU" as the primary cause and redirects the investigation toward disk or network I/O.

Narrow down which process is actually blocked, sorted by I/O-relevant state:

ps -eo pid,ppid,user,stat,wchan:20,cmd | awk '$4 ~ /D/'

    PID    PPID USER     STAT WCHAN                CMD
   2214       1 postgres D    io_schedule          postgres: writer process

STAT of D (uninterruptible sleep) combined with WCHAN (the kernel function the process is blocked inside — here, io_schedule) confirms this specific process is stuck waiting on the storage layer, not idle by choice.

Conclusion. The load spike is I/O-bound, centered on the database's writer process. The next steps — checking iostat -x 1 for device-level saturation, checking whether a backup job or a large query started at the same time, or checking disk space with df -h for a nearly-full volume that can cause pathological write behavior — follow directly from this one wa-first diagnosis, rather than from guessing based on the process list alone.

Common Mistakes

Treating Load Average as a CPU Percentage

A load of 8.0 is meaningless without knowing the core count. Always divide by nproc before judging severity, and always check wa before concluding the bottleneck is CPU rather than I/O.

Reacting to the 1-Minute Load Number Alone

A brief 1-minute spike that is already falling in the 5- and 15-minute figures is often a transient burst, not an ongoing incident. Compare all three numbers before paging anyone.

Sorting by %MEM Without Understanding Shared Memory

Multiple processes (for example several nginx worker processes, or forked application workers) can each report a similar, large %MEM for pages they actually share, not each hold privately. Summing %MEM across such processes overstates real memory pressure — see Process Memory for the accounting that avoids this trap.

Killing the Top CPU Consumer on Reflex

The process with the highest instantaneous %CPU in top is not automatically the cause of a problem — it might simply be doing legitimate, expected work (a backup, a scheduled batch job) that happens to coincide with the alert. Confirm against what should be running before terminating anything.

Forgetting That top's Snapshot Interval Hides Short Spikes

By default top refreshes every three seconds; a process that spikes to 100% CPU for half a second and then sleeps can be nearly invisible in the interactive view. For catching short bursts, use pidstat 1 (from the sysstat package) or increase sampling frequency with top -d 0.5.

Exercises

1. Run uptime and nproc together and compute whether your system's current 1-minute load average represents under- or over-subscription relative to its core count.
2. Start a CPU-bound loop (yes > /dev/null &) and a disk-bound one (for example dd if=/dev/zero of=~/lab/testfile bs=1M count=2000 oflag=direct &, inside a disposable lab directory) at the same time, then use top's CPU breakdown line to distinguish which kind of load each one contributes. Clean up the test file afterward.
3. In htop, switch to tree view and identify the parent-child relationship of a multi-process service running on your system (or start python3 -m http.server and inspect its own process tree if nothing else is available).
4. Using ps -eo pid,stat,wchan:20,cmd, find a process currently in D state on a busy system (or explain, if none exists, what command and circumstance would put a process into that state).

Verification criterion: your load-average interpretation must reference the core count explicitly, and your I/O-versus-CPU diagnosis must cite the specific %Cpu(s) field (wa versus us/sy) that supports your conclusion — not just "it felt slow."

References

• Linux man-pages: top(1): https://man7.org/linux/man-pages/man1/top.1.html
• htop project documentation: https://htop.dev/
• Linux man-pages: proc(5) — /proc/loadavg, /proc/stat: https://man7.org/linux/man-pages/man5/proc.5.html
• Linux kernel documentation: CPU load accounting: https://www.kernel.org/doc/html/latest/admin-guide/pm/cpuidle.html
• Brendan Gregg: Linux Load Averages — https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html
• Linux man-pages: pidstat(1): https://man7.org/linux/man-pages/man1/pidstat.1.html
• Linux man-pages: iostat(1): https://man7.org/linux/man-pages/man1/iostat.1.html