Embedding models benchmark for code duplication detection

Contents

Embedding models benchmark for code duplication detection

Published: August 20, 2026

Introduction

This research is supplementary material for my code duplication detection tool, Slopo. The goal is to provide data for answering two major questions:

What kind of duplicate embedding models are able to detect?

Which model is best for this task?

The script used for analysis, all code snippets, and more technical details are available on the dedicated GitHub repository. You can run and verify it yourself, add more models, adapt, and include it in your research work.

Result data are available on separate pages for easier interactive evaluation:

Gaps between cases - gap between high and low similarity cases, and gaps between single cases focusing on one aspect of similarity.

Similarity per case - average cosine similarity for each model on each test case.

Score consistency - standard deviation showing how consistently each model scores across pairs within a case.

Model labels contain the model name with dimensions used in the test. Labels with the | suffix are variants explained in the article.

Results summary

Winners:

Jina AI - jina-code-embeddings-0.5b with task code2code.query (available via API) and jina-embeddings-v2-base-code (small model for running locally).

Voyage AI - voyage-4-large (general-purpose model), but not voyage-code (both 3 and 4 scored poorly).

Use 256 dimension vectors.

Models from OpenAI, Google Gemini, Cohere are less suitable for use in code duplication detection tools.

Methodology

Models are evaluated on how good they are at separating duplicated code from not duplicated, including adversarial cases.

The data set includes 19 cases, each containing 10 code pairs, giving 190 data points per model. Pairs within a case vary by language, domain, algorithm, problem, etc. For each case, the average cosine similarity and the population standard deviation are reported.

13 cases expect high similarity and 6 low similarity. This split reflects what we consider a duplicate. Note that this benchmark doesn't use the Type-1/2/3/4 clone taxonomy from the code-clone literature.

For each model, average similarity for high and low cases is calculated separately. The gap between them gives a number used for comparing models. In practice, we want our tool to report similar code snippets above the configured threshold, discarding everything below, so the wider the gap, the more precise the results.

Cosine similarity should not be compared between models directly, because their effective ranges differ, which is visible in the data.

Cases design

The cases are designed to evaluate a single aspect, keeping other variables unchanged. Evaluation of what code does is isolated from evaluation of surface similarity. A good model encodes the code's behavior in the vector, while weak ones can only compare text, and this benchmark separates them.

A few interesting examples:

The high/different-algorithm case includes code pairs solving the same problem with different algorithms. Nothing except language syntax is shared. No shared boilerplate, different unrelated variable names, different control flow.

The high/code-and-text case directly targets the model's understanding of code. It compares code with a description in plain English. Behavior is described instead of data structures or control-flow mechanisms so it fits the duplicate detection task. No shared vocabulary, texts don't mention variable names. The snippets are not well-known algorithms that the model could easily recognize.

The low/same-boilerplate case evaluates how recognizable framework boilerplate influences similarity. The non-boilerplate part is completely unrelated, no shared vocabulary, domain, problem.

The low/unrelated case is a base case telling how low the model can score when the only similarity is the same programming language and similar code size. No shared vocabulary, domain, problem.

Adversarial cases

Some cases are designed to penalize reliance solely on surface similarities.

For example, the low/same-identifiers case (unrelated code with overlapping variable names) is a low similarity case. Reported similarity will be high since much of the text is the same, narrowing the gap between high and low cases. The model that gets tricked less (better one) reports lower similarity on this case, increasing the gap.

How it differs from MTEB

This benchmark is more focused than massive. Each case is a task intended to evaluate one aspect of duplicate detection. Fewer code snippets are used, but each is carefully designed and manually reviewed.

Model selection

The selection criteria were: include code-focused models and big popular providers used widely in the USA and Europe. Access to the model should be possible without having to deploy it myself on expensive infrastructure.

That's why I included OpenAI, Google Gemini, Cohere, and Voyage AI,...