Lab: From "Disk Is Full" to a Permanent Fix
Disk Usage and Disk-Full Troubleshooting covered df/du and the checking procedure for a full filesystem. This lab fills a real filesystem on purpose, walks the full diagnostic sequence to find the actual cause rather than the first large file you happen to notice, and then applies two different fixes depending on the constraint discovered — a genuinely full filesystem versus one that has run out of inodes — because the correct fix differs completely depending on which one it actually is.
Goal and End Result
By the end of this lab you will have diagnosed and resolved two independent disk-space emergencies on a lab filesystem — one caused by a small number of very large files, one caused by exhausting available inodes with a huge number of tiny files — using only the standard df/du/find toolchain, correctly distinguishing which fix each situation actually needs.
Topology and Starting State
A single lab VM with a dedicated, separate filesystem to fill — using a loopback-mounted image file keeps this lab fully isolated from your VM's real root filesystem, which is the safe way to practice filling a disk to 100% without any risk to the machine itself.
Requirements
- root or sudo access;
- roughly 200 MB of free space on the host to create the lab's loopback image.
Security and Snapshot Note
Warning
This lab intentionally fills a filesystem to capacity. Never do this on your real root filesystem (/) — a genuinely full root filesystem can prevent logging in, prevent sudo from working, and prevent the very commands needed to fix it from running. This lab uses a dedicated loopback filesystem specifically so a mistake has zero consequence for the host.
Step 1: Create an Isolated Lab Filesystem
sudo mkdir -p /mnt/disklab
sudo dd if=/dev/zero of=/root/disklab.img bs=1M count=200 status=progress
sudo mkfs.ext4 /root/disklab.img
sudo mount -o loop /root/disklab.img /mnt/disklab
dd if=/dev/zero of=/root/disklab.img bs=1M count=200 creates a 200 MiB file of zero bytes; mkfs.ext4 formats it as an ordinary ext4 filesystem, and mount -o loop mounts that file as if it were a real block device — a completely disposable, isolated 200 MiB disk with no connection to anything else on the host.
Verify:
Case 1: A Small Number of Very Large Files
Symptom:
Diagnose, following Disk-Full Troubleshooting's checking procedure:
The cause is immediately visible and unambiguous: one file accounts for nearly the entire filesystem's used space.
Fix and verify:
Case 2: Inode Exhaustion — Full at 1% Space Used
This is the case that surprises most administrators the first time they encounter it: df -h can report plenty of free space while the filesystem is completely unable to create a single new file.
sudo bash -c '
for i in $(seq 1 200000); do
touch "/mnt/disklab/tinyfile_$i" 2>/dev/null || break
done
'
Symptom:
The exact same error message as Case 1 — but check df -h first, before assuming the cause is the same:
Only 7% of space is used, yet the filesystem refuses to create a new file — a strong, specific signal to check the other resource a filesystem can exhaust independently of raw byte usage: inodes.
IUse% 100% confirms it precisely: every inode this filesystem was formatted with has been consumed, each by one of the 200,000 tiny files just created — as Disk-Full Troubleshooting explains, an inode is required to track a file's metadata regardless of how small the file's actual content is, and a filesystem formatted with relatively few inodes for its size (common on filesystems expected to hold few, large files) can run out of them long before running out of raw space.
Fix — removing the files frees the inodes, since each tiny file was consuming exactly one:
Interview angle: "df says 90% free — why is a write still failing?"
This is a direct, practical test of whether a candidate understands that a filesystem has two independent, separately exhaustible resources: space and inodes. The correct diagnostic reflex the moment "No space left on device" appears despite df -h showing free space is df -i, immediately — not re-running du looking for a large file that, as this lab demonstrates, may not exist at all.
Interview angle: "df says the disk is full, but du finds nothing — where did the space go?"
This is the other classic disk-full curveball, and it has a specific cause and a specific fix. When df reports a filesystem nearly full but du walking the same tree accounts for far less, the space is almost always held by a deleted-but-still-open file: a process (commonly a log-writer whose logfile was rm'd or rotated without the process being signalled to reopen) still holds the file descriptor, so the kernel keeps the data on disk — invisible to du, which only sees files still linked into a directory, but fully counted by df. The diagnostic is lsof, not du:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
rsyslogd 812 root 7w REG 8,1 4198400000 0 131 /var/log/app.log (deleted)
The fix is not to hunt for a file to delete — the file is already deleted — but to make the holding process release the descriptor: restart it (systemctl restart rsyslog), or signal it to reopen its logs, and the space is reclaimed instantly. This is exactly why a rm on a large active logfile appears to free nothing until the writing service is restarted.
Practical Scenario: A Permanent Fix When the Filesystem Genuinely Needs More Room
Problem. Unlike Cases 1 and 2, sometimes the filesystem is correctly, legitimately full — there is no single large file or inode leak to remove, the workload has simply outgrown the space it was given. On a filesystem backed by LVM, the correct fix is extending it, not searching for something to delete.
Check current state:
VFree confirms 5 GiB of unallocated space still exists in the volume group — the prerequisite for extending any logical volume within it, exactly as LVM: Resizing and Snapshots covers.
Command — extend the LV and grow the filesystem to match:
Verify:
Result. The filesystem's total size increases immediately, with the application's data intact and no downtime — the LV was extended live, without unmounting.
Conclusion. The three cases in this lab together cover the full decision tree for "disk is full": a large file to remove (Case 1), an inode leak to clean up (Case 2), or — when neither applies — a legitimate need for more space, resolved with lvextend rather than continuing to search for something that was never actually the problem.
Troubleshooting Tips
df -handdu -sh /report noticeably different totals for the same filesystem. As Disk Usage explains, a deleted-but-still-open file (held by a running process) is counted bydfbut invisible todu, sinceduonly sees what is still linked into the directory tree.find -deleteruns slowly against 200,000 tiny files. This is expected — deleting a very large number of small files is genuinely slower per-file than deleting one large file of equivalent total metadata overhead; this is itself worth knowing when estimating cleanup time during a real incident.resize2fsreports the filesystem is already at maximum size afterlvextend. Confirmlvextendactually reported a size increase (lvsagain) before assumingresize2fsfailed — a common mistake is runningresize2fsagainst the wrong device path.
Rollback and Cleanup
Final Assessment Criterion
This lab is complete when you can, from memory:
- name the exact command that distinguishes a space-exhaustion cause from an inode-exhaustion cause;
- explain why
df -halone was insufficient to diagnose Case 2; - state the two
lvextend/resize2fscommands needed to grow an LVM-backed filesystem live, without unmounting it.
Exercises
- Recreate Case 2, but this time use
ncdu(installing it first if needed) to explore the filesystem visually, and note whetherncduon its own reveals the inode problem, or whetherdf -iis still a required separate step. - Format a fresh loopback image with a deliberately small inode count using
mkfs.ext4 -N 500 <image>, and observe how much faster inode exhaustion occurs compared to this lab's default formatting. - Research and explain, in a short paragraph, why XFS filesystems handle inode allocation differently from ext4, and what practical consequence this has for the "inode exhaustion" failure mode demonstrated in this lab.