GitHub - boldizsarszakacsbekesi/diskforge · GitHub
/" data-turbo-transient="true" />
Skip to content
Type / to search
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
boldizsarszakacsbekesi
diskforge
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
master
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>5 Commits<br>5 Commits
DiskWriteCpp
DiskWriteCpp
OriginalApproach
OriginalApproach
docs
docs
.gitattributes
.gitattributes
.gitignore
.gitignore
DiskWriteCpp.sln
DiskWriteCpp.sln
LICENSE
LICENSE
README.md
README.md
View all files
Repository files navigation
diskforge
What creating thousands of files on NTFS actually costs — and what happens when you stop paying it per file.
diskforge writes files straight onto an NTFS volume by hand-building the MFT records and the directory index, without ever calling CreateFileW. It's a proof of concept, not something to run on a drive you care about — see Warnings.
I built it in a day with basically zero prior NTFS knowledge, as a way to test an idea. Shipping it as-is, thinking included.
Why I built it
I was copying a Python project and some images off a USB stick, and the copy crawled. Turned out I'd been dumb and copied the whole .venv instead of just requirements.txt, and a node_modules from another folder. Thousands of tiny files. The bytes were nothing; the count was killing it.
That got me thinking about what NTFS actually does on every file. It keeps its records in $MFT and a pile of hidden metadata files, and each CreateFileW touches all of that — allocate a record, update the directory index, write to the journal. So if I'm dropping thousands of small files, I'm paying that tax thousands of times.
So the guess was: what if I write all the metadata once, in bulk, assuming where the data is going to land, and then just drop the data onto that raw spot? Turns out it works.
The numbers
5,000 small text files into a fresh directory on the same USB volume (G:), same file count, same content, same target both times:
Approach<br>Time<br>Rate
CreateFileW + WriteFile + CloseHandle (standard Win32)<br>113.073 s<br>44 files/sec
diskforge — raw sector writes, batch-built MFT + index<br>4.854 s<br>1030 files/sec
That's about 23x , and it isn't faster I/O — both write the same bytes to the same disk. It's from paying NTFS's per-file bookkeeping (MFT record allocation, index insertion, journal transaction) once for the whole batch instead of once per file.
chkdsk G: /f reports no errors afterward, and the files open normally in Explorer (shown above — 5,000 items in BULKDIR; Explorer rounds each file up to 1 KB in the size column, but the actual data is a few bytes, stored resident in the MFT record).
Why it's fast
Every CreateFileW on a directory that already holds thousands of entries pays a real, repeated cost: allocate an MFT record, insert a key into the directory's $I30 index (which may split nodes), journal the transaction. Do that 5,000 times and the bookkeeping dominates — the one-byte write itself is noise.
diskforge computes the whole on-disk result up front — every MFT record, the full directory index, the index blocks — in memory, and writes it out in one pass. Nothing gets inserted into a tree that already has thousands of neighbors. The tree is built once, already balanced, from the bottom up.
This won't make your everyday file copies faster. It's a demonstration of where the cost actually lives.
How it works
Locks and dismounts the target volume (needs admin rights).
Extends $MFT in place to hold N new file records, updating $MFTMirr and $MFT:$BITMAP to match. (Extending $MFT rewrites its own record, which is why $MFTMirr has to change too.)
Builds one FILE record per file by hand — $STANDARD_INFORMATION, $FILE_NAME, a resident $SECURITY_DESCRIPTOR, and resident $DATA.
Bulk-builds the directory's $I30 index bottom-up (pack the leaves, promote separators, recurse) instead of inserting entries one at a time.
Writes everything out in batched sector writes, then unlocks the volume.
A couple of notes for anyone who knows NTFS well enough to squint at this:
I copied a resident $SECURITY_DESCRIPTOR from the volume root onto each file. Modern NTFS normally shares descriptors in $Secure and points to them by security ID; embedding a resident one is the older style, but it's self-contained and keeps chkdsk quiet — a deliberate PoC shortcut, not an accident.
I call the $I30 structure an "index" above on purpose. Microsoft's docs call it a B-tree; it behaves like a B+-tree with all the file entries living in the leaves. Either way, the point is it's built once instead of...