Matthew Lam on X: "introducing OpenBench v1, an open framework for measuring AI performance and efficiency for your codebase and use case.

Companies are realizing that you can't simply tokenmaxx, and are frantically looking for better ways to use and measure AI use and efficiency. One approach is https://t.co/77hMhwK498" / X<br>Post

Log inSign up

Post

Matthew Lam

@mattlam_

introducing OpenBench v1, an open framework for measuring AI performance and efficiency for your codebase and use case.

Companies are realizing that you can't simply tokenmaxx, and are frantically looking for better ways to use and measure AI use and efficiency. One approach is evidenced by the rise in model routers: OpenRouter, Cloudflare, Databricks, Vercel, and now Ramp to name a few.

But they'll also need the ability to evaluate how agents are performing in their actual use cases and codebase. A good example is @DoorDash's recent evals on their codebase and prs. This will become even more important as companies explore different model + harness combinations.

OpenBench will be built to this direction, focusing on the cross between correctness vs efficiency in both token use and latency. Starting off, OpenBench makes it easy for anyone to add to the task set, and run a variety of harness + models. The framework also makes it easy to add any custom harness variant, for example I've been testing codex variants with ablations against the stock codex harness. OpenBench will also have tooling to help with discovering/adding reliable tasks from your repo.

Today I have integrated harnesses: codex, claude, cursor, devin, grok build, pi, and eval'ed them for different models like gpt 5.6-sol, GLM 5.2, Kimi K3, Grok 4.5, etc. measuring correctness %, token use (in/out/cache), and latency.

00:00

span:not(:empty)~span:not(:empty)]:before:content-['·'] [&>span:not(:empty)~span:not(:empty)]:before:px-1 [&>span:not(:empty)~span:not(:empty)]:before:shrink-0">4:32 PM · Jul 21, 202632KViews

36036<br>24024<br>3760376<br>2940294

Read 36 replies

New to X?<br>Sign up now to get your own personalized timeline!<br>Sign up with GoogleSign up with AppleCreate account<br>By signing up, you agree to the Terms of Service and Privacy Policy, including Cookie Use.

Relevant people<br>Matthew Lam@mattlam_Follow

Don't miss what's happening<br>People on X are the first to know.

Log inSign up

Trending now