A Finite Inventory for the Linux Inode - Loke.dev

I once watched a seasoned sysadmin stare at a monitor for twenty minutes, convinced the hardware was lying to him. He had a 1TB volume with roughly 400GB of free space according to every standard monitoring tool, yet every time he tried to touch a simple text file, the shell barked back: No space left on device. It’s the kind of error that makes you question your sanity, or at least your understanding of basic arithmetic.<br>The culprit wasn't a ghost in the machine or a failing disk. It was a simple case of running out of slots in a finite inventory. In the world of Linux filesystems—specifically those in the extended filesystem family like ext4—you don't just buy storage capacity; you buy a fixed number of index nodes, or inodes .<br>The Anatomy of an Inode<br>To understand why the disk "lied," we have to look at what happens when you create a file. On a Linux system, a file is not a single cohesive entity. It is a collection of data blocks scattered across the disk, and a single metadata structure that keeps track of them. That structure is the inode.<br>The inode is essentially a record in a database. It contains almost everything about a file except for two things: the actual data inside the file and the filename itself.<br>If you run stat on a file, you can see exactly what the inode tracks:<br>$ stat example.txt<br>File: example.txt<br>Size: 1024 Blocks: 8 IO Block: 4096 regular file<br>Device: 801h/2049d Inode: 131075 Links: 1<br>Access: (0644/-rw-r--r--) Uid: ( 1000/ user) Gid: ( 1000/ group)<br>Access: 2023-10-27 10:00:00.000000000 -0400<br>Modify: 2023-10-27 10:00:00.000000000 -0400<br>Change: 2023-10-27 10:00:00.000000000 -0400<br>Birth: -<br>In this output, Inode: 131075 is the unique identifier for that file on that specific filesystem. The inode holds the permissions (0644), the owner (UID 1000), the size, and the timestamps. Crucially, it also contains pointers to the data blocks on the physical disk where the content of example.txt actually resides.<br>The filename, oddly enough, lives in a "directory file"—a special type of file that maps a string (the name) to an inode number. This is why you can have multiple hard links to the same file; they are just different names pointing to the same inode index.<br>The Fixed Inventory Problem<br>In common filesystems like ext4, the number of inodes is determined at the moment the filesystem is created. When you run mkfs.ext4, the utility looks at the total size of the partition and applies a formula—usually one inode for every 16KB of space—to decide how many inodes to bake into the disk's metadata tables.<br>Once that filesystem is mounted, that number is generally set in stone. If you have 10 million inodes, you can have exactly 10 million files. It doesn't matter if those files are 1GB each or 1 byte each. If you create 10 million 1-byte files, you will use up every single inode while leaving 99% of your disk capacity sitting empty and unusable.<br>To see this in action, you can use the -i flag with the df command:<br>$ df -ih<br>Filesystem Inodes IUsed IFree IUse% Mounted on<br>/dev/sda1 6.0M 5.9M 100K 99% /<br>tmpfs 1.9M 1.1K 1.9M 1% /dev/shm<br>In the example above, /dev/sda1 is in critical condition. Even though df -h might show plenty of gigabytes remaining, the IUse% at 99% means the system is about to hit a wall.<br>Why Does This Happen in the Real World?<br>You might think, "Who creates millions of tiny files?" The answer is: almost every modern web application.<br>I’ve seen inode exhaustion happen most frequently in three scenarios:<br>1. PHP Session Files: Older configurations of PHP stored session data in /var/lib/php/sessions. If the garbage collection cron job fails or is misconfigured, every single visitor to your site generates a tiny file that never gets deleted. Over six months, this adds up to millions of zero-byte files.<br>2. Proxy Caches: Tools like Nginx or Squid cache content in small fragments. If the cache keys are many and the expiration is long, the inode table fills up fast.<br>3. Build Artifacts and Node Modules: If you have a CI/CD runner that doesn't clean up after itself, the sheer volume of small files in node_modules folders can eventually choke the filesystem metadata.<br>Hunting the Inode Hogs<br>When the disk is "full" but not actually full, you need to find where the millions of files are hiding. The standard du -sh command won't help much here because it measures byte size, not file count.<br>Instead, you can use a combination of find, cut, and sort to get a count of files per directory. This little one-liner is a lifesaver:<br>$ find / -xdev -type d -print0 | while IFS= read -r -d '' dir; do<br>echo "$(find "$dir" -maxdepth 1 | wc -l) $dir";<br>done | sort -rn | head -20<br>Breakdown of what's happening here:<br>- find / -xdev -type d: Look for all directories, but -xdev ensures we don't cross into other filesystems (like /proc or mounted network drives).<br>- wc -l: Counts the number of entries in that specific directory.<br>- sort -rn | head -20: Shows you the top 20...