mmap vs pread in a real Go storage engine | Internals for Interns<br>Guest post<br>Introducing my guest Phuong Le<br>GitHub
VictoriaMetrics Blog
I couldn’t be happier to have Phuong Le as the very first guest of this blog.<br>Phuong is a software engineer at VictoriaMetrics<br>and one of those rare writers who can make Go internals feel simple. If you’ve<br>ever wondered how something in Go really works under the hood, chances are<br>you’ve already landed on one of his<br>outstanding posts on the VictoriaMetrics blog<br>like his fantastic series about the sync package<br>And if that wasn’t enough, he’s also the author of<br>The Anatomy of Go<br>, a deep dive<br>into the language’s type system, runtime, memory management, and concurrency<br>model. In other words: exactly the kind of person you want explaining internals<br>here. Enjoy the post!<br>When you build a storage engine in Go, sooner or later you need to answer a very plain question:<br>“How should the code read bytes from files?”<br>This sounds too low-level to matter. A database has bigger ideas: partitions, blocks, indexes, filters, compression, compaction, caches, query planning. But all of these ideas end up doing the same simple action many times:<br>“Read N bytes from file F at offset O.”<br>We will look at that action through a real codebase: VictoriaLogs<br>, a high-performance log database written in Go, and the shared VictoriaMetrics filesystem layer it uses. The point is not to sell VictoriaLogs. The point is to study a practical Go storage engine and learn why the answer is not just “use mmap” or “use pread”.<br>The boring answer is:<br>Use an abstraction. Prefer mmap when the data is already in memory. Fall back to pread when touching mmap memory could block a Go runtime thread on a major page fault.<br>That answer sounds strange if you are new to filesystems. Let’s build up to this answer together.<br>Aliaksandr Valialkin, CTO of VictoriaMetrics, wrote about this in “mmap may slow down your Go app”. The article explains that a goroutine touching cold mmap data can occupy an OS thread while the kernel resolves the major page fault, and if enough goroutines do this at once, Go execution can stall. See: valyala.medium.com/mmap-in-go-considered-harmful-d92a25cb161d
Random reads everywhere<br>A storage engine does not usually read one giant file from start to end. For a query, it may need:<br>a small block of index data<br>a small column header<br>one bloom filter<br>one compressed timestamp block<br>one values block for one field<br>another values block for another field
Those reads are not always adjacent. They are often small, often random, and often concurrent (happening in many goroutines at once). VictoriaLogs has exactly this shape. A stored data part has multiple files: index data, column headers, timestamps, bloom filters, and values.<br>We are not here to understand the VictoriaLogs structure; it deserves its own post. The important thing is that the data part keeps these files behind a common random-read interface.
The interface is not “open file and stream it”. It is “read this exact range”. That is why pread and mmap enter our design.<br>What pread does<br>pread means “read from this file descriptor at this offset”.<br>The Linux manual says pread() reads from a file descriptor at a given offset and does not change the file offset. This makes it useful for multithreaded programs because multiple threads can read from the same file without fighting over a shared seek position.<br>See the Linux manual page for pread(2): man7.org/linux/man-pages/man2/pwrite.2.html
In Go, os.File.ReadAt gives similar semantics at the Go API level. Under the hood, on Unix-like systems, this maps to positioned file reads. The shape is simple.<br>The program asks the kernel for bytes at an offset.<br>The kernel gets the data from the page cache or from storage.<br>The kernel copies the bytes into the buffer owned by the program.<br>The call returns.<br>Notice what this involves. The program enters the kernel through a syscall, the kernel copies the bytes into the buffer the program owns, and if the data is not in the page cache the call waits for disk or network storage before it returns.
So pread has visible cost. It is worth putting a number on it, because the number is what makes the whole mmap story matter.<br>A syscall is not a function call. It is a hardware-mediated crossing from user space (ring 3) into the kernel (ring 0) and back:<br>the CPU switches stacks and page tables,<br>saves and restores registers,<br>and on hardware affected by Spectre and Meltdown runs a set of mitigations that fire unconditionally on every single entry and exit.<br>Jesús Espino takes that journey apart step by step in System Calls<br>, and it is worth reading to see how much machinery surrounds one read().<br>None of that machinery is free. A bare mode switch costs a few hundred nanoseconds on modern hardware; once you add the Meltdown/Spectre mitigations...