Blazing Fast pandas Without Changing a Line — Research | doubleAI
Skip to main content<br>WarpSpeed reaches the Speed of Light on Blackwell<br>Read the Post<br>Close Announcement Banner
All Posts
Blog<br>Blazing Fast pandas Without Changing a Line<br>WarpSpeed completely rewrote pandas’ kernels from scratch, specialised for modern hardware.
Published on<br>July 26, 2026
Share this post
<br>WarpSpeed completely rewrote pandas’ kernels from scratch, specialised for modern hardware.<br>The result is a drop-in replacement version of pandas that runs 38.4x faster on a standard benchmark; reducing 10 minutes of work down to merely 17 seconds . All while retaining the same API on the surface, same semantics underneath, and remaining 100% compliant.<br>Read on to see how.
Results on PDS-H : WarpSpeed pandas against stock pandas across all 22 queries, timed on a c7i.metal-24xl with compute time only, reported as the median of 5 runs at SF=10 (10 GB).
The Benchmark<br>The standard benchmark behind that number is PDS-H : a suite of 22 queries over a mock wholesale company, the kind of business pipeline that runs on pandas daily. Each query starts life as an everyday business question. Let’s take a simple example. What were the top 5 product categories by revenue last quarter? Here is the pandas it turns into:
The real PDS-H queries get more involved than this, but their commonality is that they are all ordinary business questions, expressed as a handful of SQL-like operations.<br>We run exactly these queries from the suite maintained by the team behind Polars, keeping the same data and the same pandas queries and swapping in only WarpSpeed underneath, without changing a single line of code.<br>A worked example: merge<br>So, where does the speedup actually come from? And why it is so hard to attain without changing the semantics of pandas at all? To help answer this, let’s walk through one operation all the way down. merge is a good candidate, since it appears in nearly every PDS-H query and exercises both halves of the problem at once.
How a merge works: each order is matched to its customer through a shared ID, so every order row comes out carrying the customer's details.
A merge (or a join, in database terms) combines two tables on a column they share.<br>You have a table of orders. Each row names the customer who placed it, but only by ID; the name, address, and signup date live in a separate customer table, one row per ID. A merge looks up each order's customer and copies those columns in, so every order row comes out carrying the customer's name.<br>In the figure above, the customer table is on the left, where the ID is unique: one row per customer, so it is the side you look into. Orders sits on the right and reuse those IDs freely. So Alice, customer 1 with two orders, shows up as ID 1 twice, and both of her rows indeed come out as saying "Alice."<br>Why pandas is slow<br>pandas runs a hash join. Under the hood that requires several separate passes over the data:<br>Factorize the keys. The key column on each side is mapped to dense integer codes (a hash, for string keys). Every key on both sides is touched.<br>Build the hash table (group in the figure below). One side's codes go into a hash table mapping each key to the row positions where it occurs, keeping a list of positions since a key can repeat.<br>Probe it (match in the figure below). Each key on the other side is looked up, producing a pair of integer arrays: for every output row, which left row and which right row it came from.<br>Gather the columns. Those two index arrays are applied to every column of both frames, one "take" pass per column, copying the selected rows into freshly allocated output arrays.
None of this runs in parallel: every step is confined to a single core. Worse, most of the work is memory-bound rather than compute-bound. The build and probe scatter through the hash table at unpredictable addresses, and the gather reads source rows in whatever order the index arrays happen to point, so the core spends more of its time waiting on cache misses than doing arithmetic. The passes don't fuse either; each one writes out an intermediate that the next reads back in. In effect we pay for a single core, stalling on memory, several times over.<br>…and how we speed it up<br>WarpSpeed fixes both. It fuses the four passes — factorize, build, probe, gather — into a single sweep, so each row is touched once instead of written out and read back, and it spreads the work across cores by radix hash partitioning the keys.<br>Dispatching on the data<br>WarpSpeed goes one step further. Rather than fixing an algorithm up front, each kernel inspects its input cheaply at runtime, and when a cheap check says a shortcut is safe, it takes it, sometimes relying on conditions that a human author would rarely think to test.¹<br>A small case makes this concrete. The pandas call query.isin(subject) tests each element of query for membership in subject. By default pyarrow builds a hash set from subject and probes it. A perfectly...