The Right Harness Is All You Need — Terminal-Bench v2.1
Theme<br>Dark<br>Light
The model roster keeps growing. Since the last post, I've added Qwen 3.6 27B NVFP4,<br>Poolside Laguna S 2.1 NVFP4 to the lineup —, most interestingly, a very specific<br>community quant of GLM 5.2 that's been the standout of this round:<br>willfalco/GLM-5.2-EXL3-TR3-3.25bpw, and finally, an update to Deepseek V4 Flash, called 0731.
I originally had this post written as "all roads lead back to GLM 5.2." Despite Poolside's Laguna S 2.1 benchmarks and Deepseek's DSV4F-0731 benchmarks, I was simply unable to replicate them. The main questionmark around both were that they were benchmarked around the makers' very specific harnesses. DSV4F-0731 was claiming a better-than-GLM5.2 benchmark list, which I found very tough to believe and simply could not replicate. I almost tossed these models into the bin for good, which was sad to me because I did over one BILLION tokens locally on DSV4F-preview. I loved that model! But, when benchmarking it, and using it, I found it to seriously under-perform even DSV4F-preview!
It wasn't all bad though, we got our PCIe switch and other hardware working:
The PCIe switch finally sitting in a machine that would cooperate — retimers and MCIO adapters/cables in place, P2P conditions intact.
I was writing about model makers overfitting to their harnesses and overfitting those harnesses to benchmarks, but then I just kept hearing the name: OMP. Yet another harness on the block that people were yapping about and I decided to give it a shot with my new love: GLM 5.2 3.25 BPW EXL3 (quite the mouthful).
And man, it sure felt good using OMP with GLM 5.2. OMP definitely helps to smooth out many issues with longer-horizon tasking with agents/LLMs. It was repeatedly one-shotting my requests! But it was using a LOT of time, turns, and tokens to do it, as compared to my personal coding harness, minion. So then I wondered, well, let's try DSV4F-0731. People still wont shut up about it online, it's gotta be somewhat decent but it is definitely being hyped by a lot of people who never said a peep about DSV4F preview.
Dang, it felt good in OMP. I was still one-shotting my problems and a TON faster. GLM 5.2 3.25 BPW runs ~70-80 tok/sec for me. DSV4F-0731 runs 300-400 tok/sec. But DSV4F-0731 wasn't immediately running fast for me, there was some issue that was causing TP=4 to be quite slow. I started off by having DSV4F-0731 first fix my tok/sec, prefill speeds, and issuess that were causing TTFT (time to first token) to be quite delayed per turn. I just could tell it wasn't snappy like DSV4F-preview was and it's basically the same model so this was confusing. OMP and DSV4F-0731 solved this, it was mostly just a backend issue, but even GPT 5.6 Sol couldn't figure it out fwiw.
Well... shoot. That means I really better benchmark DSV4F-0731 with OMP huh. After doing that, I realized I needed to bench GLM 5.2 3.25 BPW too on OMP. So I did that too.
A bit about this model:
KL-divergence vs. full precision · lower = closer to reference
WeightsKV formatKLD
NF3Dynamic NVFP4 + RoPE80.139036 ± 0.002010 (n=3)<br>NF3Standard FP8 + BF16 RoPE0.1263 †<br>EXL3 3.0-bpwDynamic NVFP4 + RoPE80.119525<br>EXL3 3.0-bpwStandard FP8 + BF16 RoPE0.102508<br>EXL3 3.25-bpwDynamic NVFP4 + RoPE80.095971<br>EXL3 3.25-bpwStandard FP8 + BF16 RoPE0.087711<br>EXL3 3.36-bpwDynamic NVFP4 + RoPE80.077767<br>EXL3 3.36-bpwStandard FP8 + BF16 RoPE0.068458
The 3.25 BPW variant in the table above is the one driving our current top score. It's<br>willfalco/GLM-5.2-EXL3-TR3-3.25bpw<br>on Hugging Face. There's also a slightly-larger sibling,<br>willfalco/GLM-5.2-EXL3-TR3-3.36bpw,<br>which is on the roadmap but not something I'm rushing to test just yet. No question the KLD is better overall for the 3.36, but<br>the 3.25 was cherrypicked for coding specifically. I believe the 3.36 is focused more on overall KLD. I am certain both are great though and I *am* curious about 3.36 too! Just only so much time and I prefer to run models both thru Terminal Bench V2.1 but also actually drive as a main model before giving my thoughts. As I write this, there's a 3.40 BPW, 3.5 BPW and probably others too FYI.
The EXL3 is short for EXLlamaV3. It's a quantization format targeted for a specific CUDA inference engine. EXL3 also allows for mixed precision.
TR3 is tree based quantization, specific to EXL3.
3.25 BPW is 3.25 bits per weight, on average. Typically, more precision = better, but also slower. These particular models are highly optimized to have more bits for the things we care about and less for the things we don't. But this is all obviously a "taste" type of thing about what you'd pick.
Okay whatever, what're the results?!
Where possible, models benchmarked are set to Max thinking.
01Model Selection
click a model to toggle it on/off across all charts below
02Headline Comparison · Pass Rate
03Category Breakdown (heatmap)
pass-rate per category — brighter = stronger
The 3.25...