Disk-Full Troubleshooting

The "13. Disk, Filesystem, LVM, and Storage" module has covered Disks and Block Devices, Partitions, Filesystem Types, Mount and Umount, fstab, LVM, RAID, Swap, LUKS Encryption, NFS, Disk Quotas, and Disk Usage as separate topics. This article is the module's final, practical synthesis: it combines all of that knowledge into a single, consistent procedure for resolving one real incident — "there's no space left on disk."

What You Will Learn

By the end of this article you will be able to:

• recall the checking sequence to follow when a "No space left on device" error appears;
• distinguish, using df -h and df -i, whether the problem is byte exhaustion or inode exhaustion;
• find large directories with du and deleted-but-open files with lsof;
• free space through journalctl --vacuum-size and log rotation;
• know when to reach for LVM, quotas, and monitoring as long-term fixes.

Why "No Space Left on Device" Is Confusing

This error can come from several entirely different causes:

• the filesystem has genuinely run out of bytes;
• the filesystem has run out of inodes (there are bytes free, but a new file can't be created);
• the filesystem doesn't look full, but deleted-but-open files are holding blocks occupied;
• a quota limit has been reached (not the whole filesystem, just one user or group).

Each requires a different fix, which is why a systematic checking procedure is needed instead of trying commands at random.

The Checking Procedure: Overview

1. df -h        → which filesystem, and how full
2. df -i        → is it an inode problem, not a byte problem
3. du/ncdu      → which directory/file is responsible
4. lsof         → are there deleted-but-open files
5. clear logs or journalctl --vacuum
6. long-term: LVM extension, quotas, monitoring

Each step is covered separately below, then combined into a single practical scenario.

Step 1: df -h — Which Filesystem, and How Full

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   50G     0  100% /

This is the primary tool covered in Disk Usage — it needs no parameters and gives an instant result. The goal is to pinpoint the problematic mount point, since every following step operates inside that directory.

Step 2: df -i — Byte or Inode Problem

df -i /

Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/sda1       3276800 3276800      0  100% /

If Use% (bytes) is low but IUse% (inodes) is 100%, the problem is entirely different: no new file can be created because inodes are exhausted, even though bytes are free. In that case, searching for a large file with du is pointless — instead, find what's generating millions of tiny files (for example, a session directory or a misconfigured cache):

find / -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head

This command counts the number of files in each directory and lists the directories holding the most.

Step 3: du/ncdu — Finding Large Directories

If bytes have run out, the next step is the methods covered in Disk Usage:

sudo du -h --max-depth=1 / 2>/dev/null | sort -rh | head

Or interactively:

sudo ncdu /

The goal is descending step by step to find the culprit directory or file.

Step 4: lsof — Deleted-but-Open Files

If the sum from du is noticeably less than the "used" size df reports, the cause is almost always deleted-but-open files (explained in Disk Usage). lsof is used to find these:

sudo lsof +L1

+L1 shows files with a link count below 1 — meaning the file has already been removed from the filesystem, but some process still holds it open.

COMMAND   PID   USER   FD   TYPE DEVICE  SIZE/OFF NLINK    NODE NAME
nginx    1234    www   13w   REG    8,1  28000000     0  123456 /var/log/nginx/access.log (deleted)

An NLINK value of 0 and (deleted) at the end of the name mean the file has already been deleted, but the nginx process (PID 1234) still holds it open through a file descriptor — so 28 MB of disk blocks are still occupied.

Alternatively, with grep:

sudo lsof | grep deleted

Freeing Space: Restarting the Process

The most reliable way to free space from a deleted-but-open file is restarting the process holding it (which closes the file descriptor, automatically freeing the blocks):

sudo systemctl restart nginx

If the service can't be restarted immediately, an alternative is forcing the process to close the file's content (carefully, only in a well-understood situation):

: > /proc/1234/fd/13

This doesn't delete the file — it only clears its content; the descriptor stays open, but the occupied blocks are freed.

Step 5: Clearing Log Files

In many cases, the most common cause of a full disk is an uncontrolled, ever-growing log file.

journalctl --vacuum-size

systemd-journald logs can grow large over time. Checking the current size:

journalctl --disk-usage

Trimming it down to a given size:

sudo journalctl --vacuum-size=200M

Or removing entries older than a given period:

sudo journalctl --vacuum-time=7d

Permanent Log Rotation

A one-time cleanup treats the symptom, not the cause. The long-term fix is confirming logrotate configuration is actually working:

cat /etc/logrotate.d/nginx
sudo logrotate -f /etc/logrotate.d/nginx

-f (force) — runs rotation immediately regardless of schedule. If log rotation is already configured but the size still grew unchecked anyway, the cause is often exactly the "deleted-but-open file" problem covered above: logrotate replaced the old file, but the service kept writing to the old file descriptor because it was never restarted or signaled (SIGHUP, or a restart) through a postrotate block in the logrotate configuration.

Step 6: Long-Term Fixes

Once the situation is resolved, to prevent the same problem from recurring:

• Extending via LVM — if the VG has free space, the partition can be grown without any repartitioning at all, using lvextend -r covered in LVM: Resizing and Snapshots;
• Disk quotas — if the problem came from one user or service growing without bound, setting a limit via usrquota/grpquota covered in Disk Quotas prevents a repeat;
• Monitoring and alerting — setting up monitoring (Prometheus + Alertmanager, Zabbix, or a simple cron script) that alerts automatically when disk usage crosses a threshold (e.g. 80%) ensures the problem is caught early, well before it reaches "disk 100% full."

Practical Scenario: /var Filled Up on a Production Server

Situation: monitoring sent an alert — the web server is returning 500 errors, and the application logs show "No space left on device." The server is production, so every check is initially run in read-only mode.

Step 1. Establish the overall state.

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/vg_data/lv_var  20G   20G     0 100% /var

/var is on its own LV (the same pattern seen in LVM Basics) and it's completely full.

Step 2. Check whether inodes are exhausted.

df -i /var

Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/vg_data/lv_var  1310720  85000 1225720    7% /var

Inodes are at 7% — the problem is bytes, not inodes. The next step is finding the large file.

Step 3. Find the responsible directory.

sudo du -h --max-depth=1 /var | sort -rh | head

19G     /var/log
600M    /var/lib

sudo du -h --max-depth=1 /var/log | sort -rh | head

18G     /var/log/app

sudo du -ah /var/log/app | sort -rh | head -3

17G    /var/log/app/error.log

Step 4. Check the file — is it genuinely 17 GiB, or is du failing to count an already-deleted file?

ls -lh /var/log/app/error.log

-rw-r--r-- 1 appuser appuser 17G Jul 25 09:12 /var/log/app/error.log

The file genuinely exists and is 17 GiB. Checking the cause with the last lines:

sudo tail -n 20 /var/log/app/error.log

The result shows the same error message repeated thousands of times — the application is logging an error it can't properly resolve, retrying in an endless loop. The logrotate configuration is also checked:

cat /etc/logrotate.d/app

The result — the configuration file doesn't exist: this application has never had its log rotated.

Step 5. Free space immediately — clear the file (without deleting it) and restart the service.

sudo truncate -s 0 /var/log/app/error.log
sudo systemctl restart app

truncate -s 0 keeps the file's inode (permissions and owner stay unchanged), only clearing its content — safer than rm followed by recreating it, since it doesn't create a conflict with the application's still-open write descriptor.

Step 6. Confirm the result.

df -h /var

Filesystem      Size  Used Avail Use% Mounted on
/dev/vg_data/lv_var  20G   2.1G   17.9G  11% /var

Step 7. Long-term fix.

• create a /etc/logrotate.d/app configuration (daily rotation, maxsize 500M, compression, a retention period for old copies);
• notify the application team about the endless error loop (the root cause; the log size is only a symptom);
• add monitoring that alerts at an 80% disk usage threshold, so the problem is caught before reaching 100% next time.

Conclusion: one "disk full" alert concealed three layers — the immediate symptom (/var full), the intermediate cause (no log rotation), and the root cause (an error loop in the application). Fixing only the first layer (freeing space) would eventually bring the problem back — all three layers needed addressing.

Common Mistakes

Deleting a File Without Identifying the Cause

Deleting a large file immediately, without checking it, is risky — it can cause data loss or destroy diagnostic information needed later. Understanding what it is and why it grew so large comes first.

Looking Only at df -h and Forgetting df -i

Concluding "everything's fine" from df -h alone, in a situation where there's disk space but a file still can't be written, completely misses an inode exhaustion problem.

Trying to "Fix" a Deleted-but-Open File with a Plain rm

If a file is already deleted (lsof +L1 shows it), running rm on it again changes nothing — as long as a process holds it open, the blocks stay occupied. The fix is restarting the process.

Treating Only the Symptom, Without Checking the Root Cause

Considering the problem "solved" once space is freed, without checking log rotation or the application's underlying bug, leads to the exact same situation recurring within days.

Exercises

1. In a disposable VM, artificially create a large file (dd if=/dev/zero of=/tmp/bigfile bs=1M count=2000), hold it open with a process (for example, tail -f /tmp/bigfile &), and rm /tmp/bigfile — compare df -h and du -sh /tmp results.
2. In the situation above, find the deleted-but-open file with lsof +L1, stop the process holding it, and confirm the space is returned.
3. Check the current journal size with journalctl --disk-usage, then trim it with journalctl --vacuum-size=100M.
4. In a test VM, create a simple configuration under /etc/logrotate.d/ (for example, for /tmp/testapp.log, daily, maxsize 10M) and force it with logrotate -f, checking the result.

Verification criterion: given a "disk full" alert, you should be able to name the checking sequence (df -h → df -i → du/ncdu → lsof +L1 → clear logs → long-term fix) from memory, in order.

Summary

The "13. Disk, Filesystem, LVM, and Storage" module started with recognizing disks (lsblk, Disks and Block Devices), moved through creating partitions and filesystems, mounting and permanently connecting them through fstab, flexible volume management with LVM, reliability and security through RAID and encryption, sharing over the network with NFS, and finally space control through quotas and disk usage. This article combined all of that knowledge into a single practical procedure: df -h → df -i → du/ncdu → lsof +L1 → clearing logs → a long-term LVM/quota/monitoring fix. A single "disk full" alert often hides a multi-layered problem, which is why finding the root cause — not just the symptom — matters.

Rather than repeating this checking sequence manually every time, it can be automated — for example, by writing a script that regularly checks disk usage and sends an alert when it crosses a threshold. That exact kind of automation is the subject of the next "14. Bash Scripting" module: starting from Bash Script Basics, moving through variables, conditionals and loops, and error handling, eventually building a complete disk-monitoring and log-cleanup script.

References

• man7.org: df(1)
• man7.org: du(1)
• man7.org: lsof(8)
• man7.org: journalctl(1)
• man7.org: logrotate(8)
• man7.org: truncate(1)
• Red Hat: Troubleshooting disk space issues
• Ubuntu Server Documentation: Logs