Disk and File System Basics

/var/log/app.log looks like an ordinary path, but its name alone does not tell you where the underlying bytes actually live. That path might point to a partition on an NVMe drive formatted as ext4, to an NFS export, to a RAM-backed tmpfs, or to a container's overlay layer. When a "disk full" alert comes in, the first job of an administrator is telling these layers apart.

This article builds the mental model that connects a storage device to a file. It does not partition a real disk with fdisk/parted, or touch a production /etc/fstab — those remain separate, hands-on topics with their own lab and their own snapshot (Mount Concepts covers the mount side; a dedicated partitioning lab is still a gap in this course, noted at the end). Most of the article only reads existing state on Ubuntu Server LTS with Bash. One section does build LVM volumes on disposable loopback files, precisely so the provisioning model can be shown safely, without requiring a spare disk. The path-handling and pipeline habits from Finding Files and Searching Text are assumed. You will learn to tell apart a disk, a partition, a volume, a file system, and a mount point; see how the kernel's virtual file system (VFS) exposes different file system implementations through one interface; understand how a path resolves down to an inode; know which question to ask lsblk, findmnt, df, du, and stat; understand where LVM, RAID, and disk quotas fit into that stack; and gather read-only evidence for a "No space left on device" error, keeping in mind that block capacity and inode capacity run out independently.

The Storage Layers Are Not Interchangeable

A simplified local storage stack looks like this:

SSD or HDD
  └─ block device: /dev/nvme0n1
       └─ partition: /dev/nvme0n1p2
            └─ file system: ext4
                 └─ mount point: /
                      └─ path: /var/log/app.log

A real server can insert extra layers between the partition and the file system — LUKS encryption, software RAID, or LVM. A network file system may have no local block device at all:

export on an NFS server → mount on an NFS client → /srv/shared → files

Because of that, do not treat the following terms as synonyms:

• disk / storage device: the physical or virtual device that holds data.
• block device: the kernel's interface to a device that is read and written in fixed-size blocks.
• partition: a range of a disk carved out by a partition table.
• volume: a logical block device that LVM, RAID, encryption, or another layer presents on top of a partition.
• file system: the on-disk format and kernel implementation that organizes files, directories, metadata, and free space.
• mount: the act of attaching a file system's tree to a specific point in the directory tree a process sees.
• mount point: the directory where that attachment happens.

One disk can hold several partitions. One file system can hold millions of files. One file system can even appear at more than one visible mount. Btrfs subvolumes, bind mounts, container overlays, and network storage all break the simple "one partition, one mount" mental model, so do not rely on it once any of those are in play.

Linux Shows One Directory Tree

Instead of separate drive letters like Windows' C: and D:, an ordinary Linux process sees a single tree rooted at /. Other file systems attach at /home, /var/lib, /mnt/data, or wherever an administrator chose to mount them.

findmnt --target /
findmnt --target /var/log

findmnt -T PATH finds which mount a given path falls under. For scripting, ask for specific columns:

findmnt -no TARGET,SOURCE,FSTYPE,OPTIONS --target /var/log

A representative result:

/ /dev/nvme0n1p2 ext4 rw,relatime

That output means /var/log is not its own mount — it lives inside the ext4 file system mounted at /. On another server, /var/log could easily be a separate logical volume or a network file system of its own.

If a mount point already had files under it before something else was mounted there, those files are not deleted, but they become hidden for as long as the mount is active. Assuming "data is gone" from an unexpected empty directory is a common and avoidable mistake; the mechanism is explored with a hands-on lab in Mount Concepts.

VFS: One Interface for Many File Systems

An application does not know the internal format of ext4, XFS, or NFS. It calls system calls such as open(), read(), write(), and stat(), and the Linux kernel's virtual file system (VFS) maps that generic interface onto whichever file system implementation actually owns the target path.

process
  │ open("/var/log/app.log")
  ▼
VFS: path lookup, permission checks, mount boundaries
  ▼
ext4 / XFS / NFS / tmpfs / procfs implementation
  ▼
block stack, network stack, or kernel-generated data

This abstraction is why cat, cp, and stat can read a disk file, a network file, or a pseudo-file system object through the same path-based interface. But not every file system offers the same guarantees: case sensitivity, permission model, timestamp precision, hard-link support, snapshot capability, and write durability all vary.

ext4 versus XFS: Why the Default Differs by Distribution

Ubuntu and Debian install with ext4 as the default root file system; most RHEL-family distributions (RHEL, Rocky, AlmaLinux) default to XFS. Both are journaling file systems — before a metadata change is written to its final location, a compact description of that change is written to a journal first, so a crash mid-write can be replayed or rolled back on the next mount instead of leaving the file system structurally inconsistent. Where they differ matters for an administration decision, not just trivia:

The practical consequence: if a volume was formatted as XFS and later needs to shrink (splitting one large volume into two smaller ones, for example), there is no in-place shrink path — the only route is to create a new, smaller XFS file system and copy the data across. Confirm the file system type before promising a shrink operation to anyone:

findmnt -no FSTYPE --target /
lsblk -f

To see which file system types the running kernel currently knows about:

sed -n '1,80p' /proc/filesystems

An entry marked nodev usually means that type does not require a block device to mount. A type appearing in this list does not mean it is mounted right now — findmnt shows the actual mounted state.

How a Path Becomes an Object

/var/log/app.log is made of components:

/ → var → log → app.log

The kernel resolves a path one component at a time. A directory entry (dentry) associates a name inside a specific directory with a file system object; the VFS keeps a dentry cache to speed up repeated lookups. The inode is the object's metadata record: its type, permissions, owner, size, timestamps, and the pointers to its data blocks.

An important detail: on a traditional Unix file system, a file's name is not stored inside its inode. A directory entry links a name to an inode number. Because more than one name can link to the same inode, this is exactly the mechanism behind hard links, covered in Hard and Symbolic Links.

Inspecting an object's metadata:

stat -c 'name=%n type=%F inode=%i size=%s blocks=%b block-size=%B mount=%m' /etc/hosts

A representative result:

name=/etc/hosts type=regular file inode=123456 size=302 blocks=8 block-size=512 mount=/

• inode is the inode number within this file system.
• size is the file's logical size.
• blocks × block-size reports how much space is actually allocated.
• mount shows which mount point the object lives under.

An inode number is not a global identifier across the whole server — reliably distinguishing two objects also requires the file system or device identifier. Overlay, network, and pseudo file systems present metadata according to their own semantics, which may not match a plain local disk.

Reading the Block and Mount Maps

lsblk: the Block Layers

lsblk -o NAME,TYPE,SIZE,FSTYPE,FSAVAIL,FSUSE%,MOUNTPOINTS

A representative result:

NAME        TYPE   SIZE FSTYPE FSAVAIL FSUSE% MOUNTPOINTS
nvme0n1     disk   477G
├─nvme0n1p1 part     1G vfat      900M     10% /boot/efi
└─nvme0n1p2 part   476G ext4      370G     16% /

TYPE=disk is the whole device, TYPE=part is a partition. The tree shows the parent-child relationship. On an LVM system you may see a lvm type; on software RAID, raid1. A blank FSTYPE does not mean "device unused" — it can be a member of another layer, swap space, or a format util-linux does not recognize.

lsblk's default columns can change between util-linux versions. In a script, name the columns explicitly with --output, and consider --json or --pairs when the output needs to be parsed.

findmnt: the Current Mount Tree

findmnt --real
findmnt --pseudo

--real filters to mounts util-linux classifies as real; --pseudo isolates pseudo file systems such as proc, sysfs, and tmpfs. Do not assume these two filters perfectly categorize every network, FUSE, or unusual storage setup for your purposes — for a specific path, findmnt -T is the more reliable question to ask.

Checking mount options:

findmnt -no SOURCE,FSTYPE,OPTIONS --target /tmp

rw means writable, ro means read-only. Options such as nosuid, nodev, and noexec affect security semantics, but their presence alone is not a complete guarantee of protection.

Checking Capacity from Three Angles

df: the File System's Own Report

df -hT /var/log

A representative result:

Filesystem          Type  Size  Used Avail Use% Mounted on
/dev/nvme0n1p2      ext4  468G   71G  374G  16% /

df reports block capacity for the file system that owns the given path, not the specific directory. -h gives human-readable, power-of-1024 units; -T shows the file system type. Avail is what an ordinary process can actually use, and it does not have to equal Size - Used exactly — file system metadata, rounding, reserved space, and storage-specific accounting all account for the gap.

Inode capacity:

df -i /var/log

Filesystem          Inodes   IUsed    IFree IUse% Mounted on
/dev/nvme0n1p2     31195136 1041542 30153594    4% /

Even with plenty of free bytes, a huge number of tiny files can exhaust inodes and stop new files from being created. When you see "No space left on device," check df -i alongside df -h, not instead of it.

du: Walking a Directory Tree

du -xhd1 /var 2>/dev/null | sort -h

• -h: human-readable units.
• -d1: sum only one level deep.
• -x: stay within this file system, do not cross into other mounted file systems.
• sort -h: sort the result by size.

Silently discarding permission errors is not a good habit for production diagnostics. The 2>/dev/null above is only for a compact demonstration; for a real analysis, keep the error stream in its own file:

du -xhd1 /var > "$HOME/var-usage.txt" 2> "$HOME/var-usage-errors.txt"
du_status=$?
printf 'du status=%s\n' "$du_status"
wc -l "$HOME/var-usage.txt" "$HOME/var-usage-errors.txt"

du counts allocated blocks under a path; df reports the file system's overall accounting. Common reasons the two numbers disagree:

• a process is still holding an already-deleted file open;
• du could not read part of the tree because of permissions;
• a mount or bind mount changes what falls inside the scope being measured;
• file system metadata, reserved blocks, snapshots, or copy-on-write layers add overhead du does not see;
• a sparse file's apparent size differs from its allocated blocks.

Treat df and du as answers to two different questions, not two competing measurements of the same thing.

When df Says Full but du Cannot Find It

The most common version of this puzzle — in an interview and in production — is a file that has been unlinked from its directory while a process still holds it open. rm removed the name, so du, which walks names, no longer counts it; but the inode and its data blocks are not freed until the last open descriptor is closed. df, which reports the file system's own block accounting, still sees that space as used. The classic trigger is a log file deleted (or rotated incorrectly) out from under a running daemon that keeps writing to the now-nameless inode.

Confirm the situation rather than guessing:

sudo lsof -nP +L1

+L1 lists open files whose link count has dropped below 1 — exactly the deleted-but-still-open case. A representative row:

COMMAND    PID   USER  FD   TYPE DEVICE   SIZE/OFF NLINK NODE NAME
rsyslogd   812 syslog   7w   REG  259,2 4294967296     0  131 /var/log/app.log (deleted)

NLINK 0 together with the (deleted) marker means those 4 GiB are held only by PID 812's descriptor 7. The space returns the instant that descriptor closes.

The safe fix is to make the process release the descriptor through its normal mechanism — systemctl reload or restart the service, which reopens a fresh file. Only when a clean restart is genuinely not possible, and you understand the process, can you reclaim the space in place by truncating through the live descriptor in /proc:

sudo truncate -s 0 /proc/812/fd/7

Provisioning Layers: LVM, RAID, and Quotas

Everything so far assumes the partition-to-file-system mapping is fixed. In production, an administrator is regularly asked to grow a volume without downtime, tolerate a failed disk, or stop one user from filling a shared file system — three separate problems, solved by three separate tools sitting at different layers of the storage stack.

LVM: A Flexible Layer Between Partitions and File Systems

The Logical Volume Manager (LVM) inserts an extra layer of indirection so a file system is no longer tied to the exact size of one partition:

physical volume(s) (PV) → volume group (VG) → logical volume(s) (LV) → file system

• A physical volume is a partition or whole disk that LVM has taken ownership of.
• A volume group pools one or more physical volumes into a single space of allocatable storage.
• A logical volume is a slice of that pool, presented to the kernel as a block device (/dev/<vg-name>/<lv-name>) that a file system can be created on, exactly like a partition.

The payoff: a logical volume can be extended by adding a new physical volume to the group later, without moving any existing data, as long as the file system on top supports online growth.

Building the stack on two 200 MiB loopback files, entirely inside a scratch directory:

mkdir -p ~/lab/lvm && cd ~/lab/lvm
truncate -s 200M disk-a.img disk-b.img
sudo losetup -fP disk-a.img && sudo losetup -fP disk-b.img
losetup -a | grep lab/lvm

losetup -a confirms which /dev/loopN device now backs each file — the exact numbers vary per system, so read them from the output rather than assuming loop0/loop1.

sudo pvcreate /dev/loop0 /dev/loop1
sudo vgcreate lab_vg /dev/loop0 /dev/loop1
sudo lvcreate -L 300M -n lab_lv lab_vg
sudo mkfs.ext4 /dev/lab_vg/lab_lv

• pvcreate writes an LVM label to each device so it can be recognized as a pool member.
• vgcreate lab_vg ... combines both physical volumes into one 400 MiB pool.
• lvcreate -L 300M -n lab_lv lab_vg carves out a 300 MiB logical volume from that pool, leaving the rest unallocated for later growth.

Verify each layer read-only before trusting it:

sudo pvs
sudo vgs
sudo lvs

Growing a Logical Volume Online

This is the operation LVM is chosen for most often — extending storage without unmounting anything:

mkdir -p /mnt/lab-lv
sudo mount /dev/lab_vg/lab_lv /mnt/lab-lv
df -hT /mnt/lab-lv
sudo lvextend -L +50M /dev/lab_vg/lab_lv
sudo resize2fs /dev/lab_vg/lab_lv
df -hT /mnt/lab-lv

lvextend -L +50M grows the logical volume itself by 50 MiB, using free space still available in the volume group. That alone does not change what the file system believes its own size is — resize2fs (for ext4) or xfs_growfs (for XFS, which only ever grows, never shrinks) has to be told separately to extend the file system into the newly available space. Forgetting the second command is a common mistake: lvs reports a bigger logical volume while df still shows the old, smaller size.

Cleanup, in reverse order:

sudo umount /mnt/lab-lv
sudo lvremove lab_vg/lab_lv
sudo vgremove lab_vg
sudo pvremove /dev/loop0 /dev/loop1
sudo losetup -d /dev/loop0
sudo losetup -d /dev/loop1
rm -f ~/lab/lvm/disk-a.img ~/lab/lvm/disk-b.img

RAID: A Different Problem — Redundancy, Not Flexibility

LVM makes storage easier to resize; it does nothing on its own to survive a disk failure. That is the job of RAID (Redundant Array of Independent Disks), typically managed on Linux with mdadm, which combines several physical disks into one array at a level below LVM and file systems:

RAID and LVM answer different interview questions: RAID is about surviving hardware failure and, for RAID 0, raw throughput; LVM is about resizing and reorganizing storage without touching the disks underneath. The two are frequently combined — an mdadm array acts as the physical volume that LVM then manages — but neither replaces the other, and neither replaces backups: RAID protects against a disk failing, not against a mistaken rm or a ransomware event.

Building and testing a real mdadm array (creating a degraded array, simulating a disk failure, running a rebuild) is enough material for its own dedicated lab and is intentionally not included here.

Disk Quotas: Limiting One User's Share of a File System

LVM and RAID control how much storage exists; quotas control how a single file system's existing capacity is divided among its users. On a shared file system such as /home, quotas stop one account from consuming all the space and starving everyone else — a standard defense on a multi-user server.

The high-level workflow, at a conceptual level:

1. the file system is mounted with quota accounting enabled (usrquota/grpquota in the mount options);
2. quotacheck scans the file system and builds the initial usage tables;
3. edquota -u <username> opens an editor to set that user's soft and hard limits, for both blocks and inodes;
4. repquota -a reports current usage against the configured limits for every file system with quotas enabled.

A soft limit can be exceeded temporarily (usually with a grace period); a hard limit cannot be exceeded at all, and further writes fail once it is hit. Setting up quotas end to end also touches /etc/fstab mount options and file-system-specific tooling in more depth than fits here — treat this as the vocabulary and workflow to recognize in an interview, and consult man quotacheck, man edquota, and man repquota before configuring quotas on a real server.

Practical Scenario: A Service Cannot Write

Problem: an application cannot create new files under /var/lib/myapp, and its log shows No space left on device. You want evidence before deleting anything in production.

1. Identify the path and its mount:

   target=/var/lib/myapp
   test -d "$target" || printf 'directory does not exist: %s\n' "$target" >&2
   findmnt -no TARGET,SOURCE,FSTYPE,OPTIONS --target "$target"
   

2. Check both block and inode capacity:

   df -hT "$target"
   df -i "$target"
   

3. Look at the largest consumers within the same file system:

   du -xhd1 /var/lib > "$HOME/var-lib-usage.txt" 2> "$HOME/var-lib-usage-errors.txt"
   sort -h "$HOME/var-lib-usage.txt" | tail -n 20
   

4. Confirm the mount has not silently become read-only:

   findmnt -no OPTIONS --target "$target"
   

5. Separate the evidence before drawing a conclusion:

   • Use% near 100%: likely a block capacity problem.
   • IUse% near 100%: likely too many small files, an inode problem.
   • ro in the options: the file system has gone read-only.
   • df large but du small: possibly a deleted-but-open file, or a file-system-specific accounting difference.
   • the du error file is not empty: the analysis may be incomplete.

sudo may be required for some of these read-only checks purely because of permissions, but do not add it reflexively. Even when you are authorized to use it, remember that sensitive directory names and sizes can end up in whatever diagnostic file you save.

Production and Distribution Notes

• Ubuntu and Debian commonly default to ext4; many RHEL-family installation profiles default to XFS. Confirm the actual system with findmnt or lsblk -f rather than assuming.
• A cloud VM's /dev/nvme... name does not guarantee a physical NVMe disk — the hypervisor may be presenting a virtual device under that name.
• lsblk shows the host's block devices, but a container's own mount view can differ from what lsblk reports on the host.
• On a network file system, df reflects the capacity and quota semantics the server chooses to report, which may not match the underlying physical storage exactly.
• /proc, /sys, /run, and parts of /dev are not ordinary on-disk directories — do not interpret their du output as real disk usage.
• Adding sync or df --sync to routine diagnostics increases I/O load; stick to the standard read-only accounting unless you have a specific reason not to.

Common Mistakes

• Using "disk," "partition," "file system," and "mount point" as if they were the same word.
• Reading a blank FSTYPE in lsblk as "empty disk."
• Drawing conclusions about an entire disk without first confirming which mount a path actually belongs to.
• Assuming inode capacity is fine just because df -h shows free space.
• Expecting du and df numbers to always match exactly.
• Believing a file's name is stored inside its inode.
• Treating a pseudo file system as if it were an ordinary directory on an SSD.
• Assuming a host and a container always see the same mount namespace.
• Hiding permission errors and treating a partial du result as complete.
• Deleting files or altering a file system before finishing diagnostics.

Exercises

1. Run findmnt -T for /, /home, and /tmp. Determine which of them share the same file system.
2. Run lsblk with exactly seven explicit columns of your choosing, and label which rows are disks, partitions, or other layers.
3. Create a test file in your home directory and use stat to explain its inode number, logical size, and allocated blocks.
4. Compare the block and inode figures from df -hT "$HOME" and df -i "$HOME".
5. Under ~/lab, compare du -hd1 and du --apparent-size -hd1. Even if the numbers match, explain which measurement each command is actually asking for.
6. Check /proc, /sys, and /run with findmnt, and note the source and file system type reported for each.

Verification criterion: for any given path, you should be able to answer "which mount, which file system, which source, and what is the block and inode status?" using only read-only commands.

References

• Linux kernel documentation: Virtual File System: https://www.kernel.org/doc/html/latest/filesystems/vfs.html
• Linux man-pages: path_resolution(7): https://man7.org/linux/man-pages/man7/path_resolution.7.html
• Linux man-pages: inode(7): https://man7.org/linux/man-pages/man7/inode.7.html
• util-linux: lsblk(8): https://man7.org/linux/man-pages/man8/lsblk.8.html
• util-linux: findmnt(8): https://man7.org/linux/man-pages/man8/findmnt.8.html
• GNU Coreutils manual: df invocation: https://www.gnu.org/software/coreutils/manual/html_node/df-invocation.html
• GNU Coreutils manual: du invocation: https://www.gnu.org/software/coreutils/manual/html_node/du-invocation.html
• GNU Coreutils manual: stat invocation: https://www.gnu.org/software/coreutils/manual/html_node/stat-invocation.html
• Linux man-pages: mount namespaces: https://man7.org/linux/man-pages/man7/mount_namespaces.7.html
• Red Hat: LVM Logical Volumes: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/configuring_and_managing_logical_volumes/index
• Linux man-pages: lvm(8): https://man7.org/linux/man-pages/man8/lvm.8.html
• Linux man-pages: mdadm(8): https://man7.org/linux/man-pages/man8/mdadm.8.html
• Linux man-pages: quotacheck(8), edquota(8), repquota(8): https://man7.org/linux/man-pages/man8/quotacheck.8.html
• XFS documentation: xfs_growfs(8): https://man7.org/linux/man-pages/man8/xfs_growfs.8.html