Worth Accepting: Evaluating Mellum at JetBrains - by Dmitry

SubscribeSign in

Worth Accepting: Evaluating Mellum at JetBrains<br>I led evaluations for the first Mellum. Most of the hard problems were not about the model.

Dmitry<br>Mar 17, 2026

Share

Every time you see a cloud completion in your JetBrains IDE, chances are you are interacting with Mellum, JetBrains’ family of open-weight LLMs for code. In 2023, the task was to build the first one: JetBrains’ own Copilot, trained and productioned entirely in-house. It shipped in 2024, small enough to be efficient but good enough to be useful for hundreds of thousands of developers worldwide. Before shipping, we needed to ensure the model is not shabby.<br>Thanks for reading Over-Nite Evaluation! Subscribe for free to receive new posts and support my work.

Subscribe

Subscribe

Mellum is definitely not a frontier model. By design, it is a fairly small 4B Llama 2-like model trained on permissively-licensed code. One distinction worth making explicit: Mellum does code completion, not code generation. Code generation turns natural language into code. Code completion turns code into more code. The model uses fill-in-the-middle (FIM): the code before the cursor becomes the prefix, the code after it the suffix, and the model generates what goes in between.

Completion of Kotlin code proposed by Mellum in IntelliJ IDEA (Source: https://blog.jetbrains.com/ai/2024/11/jetbrains-ai-assistant-2024-3/)<br>The problem space is narrower than generation, which means we can lean on context (imports, type definitions, surrounding functions) to evaluate quality with higher confidence. I was responsible for model evaluations, which is a good place to sit: close enough to know the subtle nuances, distant enough to see the big picture.<br>Mellum2 is out now; I was not involved in it. What follows is what we learned building the first one, and most of it should apply to whatever comes next.<br>Separating Rejection from Selection

The most important strategic decision in our evaluation process was keeping two jobs clearly apart: offline evaluation existed to reject bad configurations , and online evaluation existed to select among the good ones . This is not a new idea. Netflix described it well back in 2013, and we followed their framework deliberately.

Netflix framework for faster innovation through offline and online feedback loops (Source: https://www.ueo-workshop.com/wp-content/uploads/2013/10/UEO-Caitlin.pdf)<br>Due to the high number of moving parts (context collection method, FIM splitting strategy, prefix and suffix trimming, stop tokens, programming language, library usage, single-line vs. multi-line completion), we needed a multi-step process where each stage was more expensive and more directly connected to real user behaviour than the last . A private offline benchmark ran on every checkpoint: scalable, fast, and easy to automate. LLM-as-a-judge and public benchmarks provided relative comparisons. End-to-end IDE runs closed the offline loop. Finally, A/B testing on real users made the call: the most informative signal, but also the most demanding to run.

A high-level overview of the Mellum evaluation pipeline<br>Traceability Is Not Optional

Most public benchmarks target generation, not completion, which our small FIM model was never designed for. Those that do target completion are rarely a good fit: wrong language mix, no guarantees about data freshness, and no connection to how IDEs actually collect context that materially impacts the quality. This is one of the strongest sides of Mellum: while we did not innovate much on the model architecture, we tailored the whole pre-training and post-training data pipelines to our in-IDE use case . We needed something we could trust, so we built JetComplete. The investment was significant, but it gave us full control over what mattered: language coverage, data freshness, consistent prefix/suffix/middle splitting, in-IDE preprocessing, and a clean validation/test split.<br>JetComplete covered the full set of languages Mellum needed to support (Java and Kotlin, JavaScript and TypeScript, Python, PHP, and others) under one set of principles, with a deliberate focus on fresh code to avoid data leaks. Code was processed with in-IDE tools and filtered through an LLM-as-a-judge step, then scored on exact match, chrF, and the KK score, a proprietary relaxed Levenshtein metric.<br>The pipeline ran on every checkpoint via Kubeflow: download the dataset, split it, run inference across GPU machines, combine the results, compute metrics with confidence intervals, upload reports. Over the course of the project, that meant thousands of checkpoint evaluations. Every number was traceable to the exact step that produced it. That lineage saved us more than once, and made me miss Yandex Nirvana, an excellent job orchestrator, even more.<br>If the pipeline needs explaining, it will not get adopted by the people. Evaluations should start automatically on each new checkpoint...