Your GPU Probably Isn't Helping Your Retrieval System | Strake | Strake<br>Skip to content

Most "just use a GPU" advice is wrong for how anyone actually runs small models.

I spent yesterday benchmarking a 33M parameter embedding model across five hardware backends. The results were not what I expected.

The setup

Model: BAAI/bge-small-en-v1.5. 33M params, 384-dim output. The workhorse small embedder a lot of retrieval systems use.

Workload: LongMemEval oracle split, 500 instances, batch=1, single-query retrieval per call. Published academic benchmark, not a synthetic microbench. The query distribution is realistic.

Three machines.

A Mac M2 Pro with 16 GB of unified memory, running Metal via PyTorch MPS.

A Linux desktop with an Intel 13700K, 62 GB of RAM, and an RTX 2080 Ti with 11 GB of VRAM. Running Ubuntu 22.04 and CUDA 13. I had to fight the driver to get there. More on that below.

A Windows desktop with an AMD 5800X, 64 GB of RAM, and an RX 6600 XT with 8 GB of VRAM. Running Windows 11 with DirectML on top.

Metric: p50 and p95 latency per embedding call, measured across 500 instances.

The numbers

BackendHardwarep50 (ms)p95 (ms)MetalM2 Pro (unified memory)10.635.0CUDA 13RTX 2080 Ti17.820.8CPUIntel 13700K22.234.6CPUAMD 5800X18.921.7DirectMLRX 6600 XT17.922.0<br>What jumps out

DirectML on the 6600 XT is statistically break-even with the AMD CPU on the same machine.

The GPU "acceleration" did nothing.

CUDA on the 2080 Ti wins, but only by about 20 percent on p50 and 40 percent on p95. That is a GPU costing five times what the 6600 XT does. The win is real but modest.

Metal wins outright. Unified memory eliminates host-to-device transfer entirely. There is no copy to make. Only dispatch overhead.

Why

At batch=1 with a 33M parameter model, you are dispatch-bound. Not compute-bound.

The per-call cost of kernel dispatch plus host-to-device transfer roughly equals the cost of just running the forward pass on a modern CPU. The GPU never gets a large enough block of work to justify its setup overhead.

This is the part of the "use a GPU" advice nobody mentions.

It is correct for big models. It is correct for big batches. It is correct for long sequences.

For small models running one query at a time, the math goes the other way.

The threshold shifts based on three things.

Model size: more params, more compute, more amortization of dispatch.

Batch size: more parallel work, same overhead spread thinner.

Sequence length: longer prompts, more matmul, same logic.

If you are embedding 32 documents at once on a serious GPU, CUDA wins decisively. If you are embedding one query at a time on a midrange consumer card, you are paying GPU tax to do CPU-scale work.

A debugging story you will probably relate to

On the Linux box, torch.cuda.is_available() returned False on a working CUDA install.

My first move was to blame the driver. It was on 535. Surely too old.

I bumped it to 580 and the problem went away.

The fix was correct. The diagnosis was not.

The actual root cause: torch 2.12.0+cu130 ships the CUDA 13 runtime. The 535 driver only supported CUDA 12.2. PyTorch needs runtime and driver to be ABI compatible. They were not. CUDA reported unavailable.

The driver was not "too old" in some general sense. It was specifically mismatched with my torch build tag.

I spent an hour writing the wrong root cause into my notes before I checked torch.version.cuda and saw the actual story.

If you are debugging a "broken" CUDA install, here is the order to check.

First, what CUDA version was your torch wheel built against. torch.__version__ tells you. cu130 means CUDA 13.

Then, what does your driver expose. nvidia-smi shows the max CUDA version it supports.

Mismatched ABI is failure mode one. Outdated driver is failure mode two. They look identical until you check.

The benchmark methodology trap

One more result caught me cleanly. Worth flagging.

Total wall time across 500 instances went up when I enabled CUDA. From 35.6 seconds on CPU to 64.6 seconds on CUDA. Even though per-call latency dropped.

The reason: my benchmark was re-initializing the model for every instance. GPU context init dominated the runtime. Per-call latency improved. Throughput regressed. CUDA looked slower in aggregate.

In production, where the model is loaded once and serves many queries, this inverts entirely.

If your benchmark structure does not match your production structure, you will measure the wrong thing. Cold-start cost is real. It lives outside the fast path.

The takeaway

For small embedders at batch=1, Metal with unified memory is fastest. CUDA is modestly better than CPU. DirectML is break-even with CPU.

The throughput answer is honestly kind of boring.

The reason is the part that generalizes. Dispatch overhead dominates at this scale.

If you are reaching for a GPU because that is what you do with ML, measure first. There is a real chance your CPU is fine and you are optimizing the wrong thing.

Rob is building...