Let's see Paul Allen's SIMD CSV parser | chunkofcoal.comLet's see Paul Allen's SIMD CSV parser<br>March 20, 2026<br>readers<br>Look at the subtle nibble extraction. The tasteful lookup tables of it. Oh my god, it even has vqtbl1q_u8 and vmull_p64.

Editor's note: it seems some readers are confused who Paul Allen is. It is a reference to the movie American Psycho. Good explanation here or here. It is not a reference to Microsoft co-founder and billionaire Paul Allen and his SIMD CSV parser.<br>A year ago I wrote a CSV parser that is able to parse 64 characters at a time. It&rsquo;s purely for research and hand waves over crucial steps for a production parser like validation. But the core algorithm uses SIMD and bitwise operations to classify and filter structural characters in bulk, and these are the techniques I&rsquo;ll be talking about today.<br>If you are new to SIMD, I would recommend pausing here and reading McYoung&rsquo;s introduction to SIMD. But here&rsquo;s a quick primer on SIMD:<br>CPU clock speeds hit a ceiling about 20 years ago. We can&rsquo;t make cores faster without melting them, so instead of processing one value at a time faster, we process multiple values at once (wider)<br>SIMD (single instruction, multiple data) lets you perform the same operation on a fixed batch of data (usually 16 or 32 bytes, or even 64 bytes) in the same time it takes to process a single byte<br>SIMD code (or vectorized code) is most effective when it&rsquo;s branchless, meaning it avoids if statements, loops, and function calls, performing the same operations regardless of input<br>Each architecture has a different set of SIMD instructions. See Rust&rsquo;s std::arch module<br>The simdjson paper<br>For a given topic, there are always a couple of standout papers that are considered required reading for that problem space. For example, as Joseph Beryl Koshakow put it, &ldquo;the Amazon DynamoDB and Google Spanner papers are among the canonical database papers that all database developers should read.&rdquo; [source]joekoshakow.com<br>He then said, &ldquo;Matthew is one of the smartest engineers I know and is much taller than me.&rdquo; [source-needed]<br>For SIMD, I would argue the simdjson paper is the paper to read. JSON parsing is a familiar problem, but simdjson solves it by scanning and processing 64 bytes at a time. If you prefer a video, Daniel LemireDaniel Lemire vs the rest of us<br>, the co-author of the paper and the LeBron James of SIMD, gave a talk about it as well.<br>The rest of the blog post will explain some of the techniques outlined in the paper that I used for my CSV parser. The SIMD instructions I use are ARM NEON, but x86 has direct equivalents for all of them.<br>CSV and thinking parallel<br>Here&rsquo;s a typical CSV:<br>nameagelocationalice30Irvinebob25123 Union Square, New York New Yorklonely charlie28where she said, "hi"<br>to meWhich, as a raw byte stream adhering to the CSV RFC4180 spec, looks like this:<br>name,age,location\r\n<br>alice,30,Irvine\n<br>bob,25,"123 Union Square, New York New York"\n<br>lonely charlie,28,"where she said, ""hi""\nto me"Every file format has structural characters , the characters that give shape to an otherwise flat sequence of bytes. In CSV, commas (,) separate columns, newlines (\n or \r\n) separate rows, and quotes (") wrap fields that contain structural characters themselves, like "123 Union Square, New York New York". To include a literal quote inside a quoted field, you double it: "hi" becomes ""hi"".<br>Parsing a CSV boils down to 3 steps. First, classify every byte:<br>name,age,location\r\n<br>alice,30,Irvine\n<br>bob,25,"123 Union Square, New York New York"\n<br>lonely charlie,28,"where she said, ""hi""\nto me"But not all of these are real delimiters. The comma inside "where she said, ""hi""\nto me" is just text, the \n is part of the field value, and the "" around hi are escape sequences, not boundaries. So we filter them out:<br>name,age,location\r\n<br>alice,30,Irvine\n<br>bob,25,"123 Union Square, New York New York"\n<br>lonely charlie,28,"where she said, ""hi""\nto me"What remains are the characters that actually separate columns and rows:<br>name,age,location\r\n<br>alice,30,Irvine\n<br>bob,25,123 Union Square, New York New York\n<br>lonely charlie,28,where she said, ""hi""\nto meFinally, we record the position of each remaining structural character. These offsets are all we need to slice the original byte stream into rows and fields.<br>The trick is that each of these steps can be performed on many bytes at once.<br>Step 1: Classify structural characters<br>A scalar approach would classify each byte by comparing it against every structural character one by one.<br>Geoff Langdalebranchfree.org<br>, co-author of simdjson and creator of Hyperscan, devised a technique called vectorized classification that can classify 16/32/64 bytes in a single pass.<br>The idea is to build a pair of lookup tables that act like a perfect hash. Every structural character maps to its class (COMMA = 1, QUOTE = 2, NEWLINE = 3), and everything else maps to 0. We could build a 256-entry...