Open-Weight LLMs Have Caught Up on Accuracy

LLMs for Engineers

SubscribeSign in

Open-Weight LLMs Have Caught Up on Accuracy<br>Introducing the ClinReg Benchmark for Life Sciences. This post walks through what we measured, what surprised us, and what it means for anyone deploying AI in clinical and regulatory settings.

Suchismita Padhy, Wenzhe Xue, Niklas Nielsen, and Arjun Bansal<br>Jul 30, 2026

10

Share

Accuracy is the whole game

In regulated life-sciences work — screening literature for post-market surveillance, extracting structured data from clinical documents and generating regulatory submissions, and generating tables, listings, and figures (TLFs) based on clinical trial data — a wrong answer isn’t just an inconvenience. A silently missed paper in a systematic review or a fabricated value in a data-extraction table can propagate into a regulatory submission. In the life-sciences context, accuracy matters most given the high stakes.<br>The prevailing assumption has been that only closed, frontier models can be trusted with tasks like these. We put that assumption to the test — and found it no longer holds.<br>Within life sciences, most benchmarking attention has gone to discovery. Structure prediction has CASP, the decades-old blind assessment that AlphaFold turned into the field’s canonical measuring stick. LAB-Bench and BixBench probe whether agents can reason over biology literature and run real bioinformatics analyses — BixBench, notably, is hard enough that frontier models scored in the single-digit-to-teens percent range at release. Coscientist showed an LLM planning and executing its own chemistry experiments on lab hardware. That work matters, but it measures the front of the pipeline. The money and the years are at the back: the Congressional Budget Office puts discovery and preclinical work at about 31% of pharmaceutical R&D spend — roughly $474M per approved drug — against about $1.07B for clinical trials, and about 95 months in the clinic versus 31 months preclinical. Other regulated professions have started measuring their own equivalent of that back half. Harvey’s Legal Agent Benchmark scores agents on more than 1,250 real law-firm assignments against expert-written rubrics; OpenAI’s HealthBench and Stanford’s MedHELM cover clinical care tasks; FinanceBench tests analyst questions against public filings. Outside any one vertical, GDPval is the closest to a general “real professional work product” benchmark, but does not cover regulatory submissions. Regulatory and clinical-trial workflows — where a life-sciences program spends most of its time and money — has had no public equivalent. ClinReg is our attempt at one.<br>“ClinReg” spans three real regulatory and clinical-trial tasks. We ran the leading open-weight and closed-weight models through it. The headline: across all three tasks, open-weight models now fall within the variance band of closed-source models on accuracy. And they do it at a fraction of the cost — the cost gap is significant, with open-weight models running 3x cheaper than the best scoring model we tested, and ~10x cheaper than the most expensive frontier models.<br>We’ll now walk through what we measured, what surprised us, and what it means for anyone deploying AI in clinical and regulatory settings.<br>Three things open-weight models buy you

For our customers, moving high-stakes workloads onto open models comes down to three value propositions:<br>Accuracy at a fraction of the cost. If open-weight models match frontier on the task that matters, the economics change completely.

Infrastructure robustness. Frontier models often run into capacity and infrastructure issues. By shifting the load to open-weight models, teams can achieve enhanced robustness and higher uptimes on their workloads.

Data sovereignty. Beyond accuracy and reliability, the open-weight model value proposition is also about data sovereignty — not leaking data to external providers.

The rest of this post is about proving the first point rigorously, because without it the second and third ones don’t matter.<br>Results: Open-weight is at parity on accuracy

Details on the ClinReg Benchmark tasks, datasets, evaluations, skills, harness, inference provider and score computation are provided in the Appendix .

We measured the “Overall Score” (see Appendix) across 19 models (12 proprietary and 7 open-weight). We found that GPT 5.6 Sol had the highest overall score (88.4; std dev=4.6) on the ClinReg benchmark, followed closely by open-weight models GLM 5.2 (87.4; std dev=5.6) and Kimi K3 (86.9; std dev=4.6). GLM 5.2 and Kimi K3 were 33.8% and 59.6% of the cost of running GPT 5.6 Sol on average per task. Thus, we see that open-weight models are now well within a standard deviation of the best performing proprietary model. Just a few months ago, the best performing models were Opus 4.8/4.7 which were ~9x the cost of GLM 5.2 on this task, indicative of the fast-pace of cost and accuracy improvements.<br>Just behind these...