Muninn: the best code localization model at any size

Sign in<br>Subscribe

Introducing Muninn and Quarry<br>Muninn is a state-of-the-art code localization model (English to function-level code chunks), and Quarry is a new benchmark measuring code localization for agentic search. Muninn clocks in at just 346M parameters, small enough to be fast even on a weak GPU, and Muninn-small (47M) is small enough to run on CPU alone. All are Apache-licensed.<br>(And yes, the bar is low, OpenAI's text-embedding-3 is two and a half years old now, but people still use it, and tiny Muninn-small beats it.)<br>Muninn and Muninn-small are built into the Bifrost code analysis tool, where they’re configured to use 512- and 384- dimension vectors, respectively, compressed to 8bit using fastrq. Add Muninn semantic search to your harness of choice with a quick uv tool install brokk-bifrost or npm install -g @brokkai/bifrost.<br>Vector Search for Agents<br>In the Old Days (~2024), everyone wanted to one-shot context retrieval because models were dumb and modern harnesses didn't really exist. So all the code search models from that era are trained + evaluated on mapping (initially) docstrings and (later) full issue text to code fragments. Both of these have the virtue of being easily available, but they come with the minor downside of just not being very good at modeling the kinds of queries that people (and agents) actually need to make.<br>Modern harnesses are designed to allow the model to make tool calls incrementally as part of task execution, modifying its plan based on the results. So we need a search model that focuses less on “here’s the ticket, find me everything related” (which includes lots of really bad issue descriptions as well as things like stacktraces that really don’t need vector search at all) and more on “Find me the code that does X.”<br>So we had DeepSeek (for Muninn) and Luna (for Quarry) first generate task descriptions from commits, then generate natural language queries from the tasks and map them to the most-relevant functions in the original commit’s changed files as well as related files (Bifrost most_relevant_files, based on co-edits and import analysis). We used the same design for both training dataset and for Quarry, from disjoint repos. (We switched models because the Luna price cuts made it a no-brainer, with the happy byproduct that we’re not just measuring “the model memorized DeepSeek’s idiosyncratic query generation.)<br>Muninn is trained on 11 languages: C, C++, C#, Go, Java, JavaScript, PHP, Python, Rust, Scala, and TypeScript. (Since the Muninn training, Bifrost has also added support for Kotlin and Ruby.)<br>Quarry: A new Benchmark<br>A good benchmark for search as used by modern agents should:<br>Be based on real repos with the entire repo’s codebase as search candidates, not just short excerpts.<br>Measure natural language queries, not docstrings, pseudocode, or issue texts.<br>Include tasks from all common languages, not just Python.<br>Use the most relevant metric in the headline. For agentic search, that’s micro recall (how many of the gold functions were retrieved), not MRR (how high the first one ranked), because the former measures what the agent sees; within reason, it is better to include false positives (the agent can ignore noise) than to leave out true positives (the agent never sees them).<br>Provide enough tasks to be able to meaningfully differentiate between “pretty good” and “best in the world”.<br>We measured Muninn on the close-if-you-squint evals, but it was clear that if we wanted an actually useful measurement we needed to build it ourselves.

APPS: algorithmic text → standalone code. Competitive-programming problem statements as queries, correct solution programs as documents. Python only.<br>CodeSearchNet: docstring → code. The only multilingual dataset (Python, Java, JavaScript, PHP, Ruby, Go), but very small, just 1k functions each. (One of the reasons it’s so saturated.)<br>CosQA: short English → code. Real web search queries ("python how to sort dict by value") matched to functions. Python only.<br>SWE-Bench-Lite localization: issue → edited-code. Full GitHub issue as the query, retrieving the functions edited by the fixing PR. Python only.<br>LocBench: issue → edited-code on a newer set of PRs set built to dodge SWE-bench contamination and broaden beyond bug fixes. Python only.<br>Quarry construction<br>To balance between eval expense (dominated by repo count, since you can substantially reuse a repo’s embeddings across tasks from that repo) and coverage/diversity, we selected 5 large repos with deep history from each of our 11 languages and generated 3,411 tasks and 6,525 queries across them. These repos were distinct from the repos that Muninn’s training data was generated from.<br>Construction of the Quarry dataset was as follows:<br>Identify “interesting” commits. (Primarily code changes, not merge commits, etc.)<br>Reverse-engineer the commit into task instructions that could be applied to the preimage, and filter out tasks too...