The Benchmark Meaning Gap - The JetBrains Blog

JetBrains Research

Research is crucial for progress and innovation, which is why at JetBrains we are passionate about both scientific and market research

About JetBrains Research

Research<br>The Benchmark Meaning Gap

Katie Fraser<br>Mikhail Evtikhiev<br>Sergey Titov

In any recent model comparison, you’ve probably seen a single number from a coding benchmark to represent “How good this model is at coding”. Those scores are tempting: they’re simple, they’re leaderboard-friendly, and they seem to tell a clear story about progress.

But that story didn’t hold up when we looked more closely at benchmarks that are considered standards. The models dramatically improved on the exact tasks we trained them on, yet those gains often did not show up on other benchmarks or on slightly different tasks in the same codebase. This blog post is about our research on that meaning gap between what coding benchmarks measure and what we wish they measured – and how to improve these benchmarks. This paper will be presented by our team at the Deep Learning for Code (DL4C) workshop, co-located with the International Conference on Machine Learning (ICML) this July in Seoul.

Specifically, we will:

Compare common benchmarks currently in use.

Present our own empirical comparison and discuss why these common benchmarks fail.

Propose solutions for better evaluation and argue for sustained benchmark maintenance.

Models and benchmarks

Before we begin with comparing benchmarks, we will first zoom out to define foundation models , or the systems being evaluated by the benchmarks. These are AI systems trained on massive datasets to acquire general-purpose capabilities rather than being specialized for a single task. Large language models (LLMs) are a type of foundation model – and arguably the most common type at that. Foundation models, however, can be any type of system, not limited to language.

Instead of building a new model for each use case, you start from a single, powerful foundation that can be steered toward chat, coding assistance, search, or domain-specific agents with comparatively little additional data. In creating the model, there is a preliminary, pre-training phase, where models are trained on large amounts of data, typically a blend of web text, documentation, code repositories, and other data sources.

Then they are adapted in the post-training phase, where a variety of approaches can be applied to customize the model. These include:

Reinforcement learning

Time-test scaling

Fine-tuning

Crucially, in the post-tuning phase, the model no longer trains on a large dataset, but rather on smaller sets of tokens relevant to the approach and overall goals. For more details, this paper offers a recent look at common post-training methods for LLMs.

Beyond training the models, it is essential for model creators and researchers to evaluate their capabilities, such as coding, common-sense reasoning, and natural language understanding, which covers tasks such as translation, question answering, and summarization. Benchmarks are central to the evaluation process because they turn these capabilities into quantitative scores that reveal where a model is strong and where it still struggles. Those scores then shape how researchers fine-tune and improve models, helping decide which training strategies are worth pursuing and which aren’t. However, benchmarks come in very different flavors, and each captures only a thin slice of “coding ability”. In the rest of this section, we show that benchmark results should be viewed with caution – and not necessarily as comprehensive evidence of a model’s coding capability.

Common coding benchmarks

Coding benchmarks differ a lot in what they actually test, even when they all claim to measure “code generation”. At one end, you have self-contained code tasks like HumanEval (GitHub), where the model sees a short problem and must produce a function that passes a handful of tests. There are adjacent examples like LiveCodeBench, which add self-repair, code execution, and test output prediction to code generation. At the other end are repository-level benchmarks such as SWE-bench (website), where the model is dropped into a real codebase and asked to resolve GitHub issues by navigating files, understanding context, and synthesizing patches that make the full test suite pass. We will now look at these in a little more detail.

Within the self-contained type of benchmarks, HumanEval is a small, function-level benchmark. Namely, it uses 164 hand-written programming problems, each with a prompt, a Python function signature, and hidden unit tests. It also focuses on short tasks such as write me a quicksort and produces corresponding reports, such as pass@k. Benchmarks such as HumanEval are great for quickly checking whether a model can correctly synthesize snippets of code from a description, but they tell you little about how that model behaves inside a...