Honey, I shrunk the embeddings: Matryoshka vs. PCA – Dylan Castillo

Subscribe to my newsletter

As people began using LLMs with their own documents, a new problem emerged: how do you store and search all that information efficiently?

Vector databases quickly became the standard solution. But vectors can contain thousands of dimensions, and storing millions of them can make retrieval slow and expensive.

AI labs responded with a technique called Matryoshka Representation Learning (MRL), which lets you use fewer embedding dimensions without sacrificing much accuracy in your retrieval. That means smaller vector database bills and faster queries.

Happy ending. Almost.

I do not usually spend my Wednesdays worrying about vector database bills. But Doug Turnbull’s article about using Principal Component Analysis (PCA), to reduce vector dimensions made me curious: how would this older, simpler technique compare with MRL?

To find out, I compared the two methods across eight standard retrieval-quality datasets. In this article, I walk through the experiment and share what I found.

All the code and data is available on GitHub.

What are MRL and PCA?

Both methods produce smaller vectors that behave almost like the full ones. But they get there in different ways.

MRL works during training. You train the model with the loss applied at several prefix lengths at once: the first 64 dimensions, the first 128, and so on. This teaches it to pack the most important information at the start of the vector, like a set of nested matryoshka dolls.

At inference time, you simply keep the first d dimensions and re-normalize the resulting vector. Many modern embedding models are trained this way, but older models aren’t, so MRL-based truncation is not available for many popular embedding models.

PCA works after training, which means it can be used with any model. You take a sample of embeddings, find the directions along which they vary the most, and keep the top d of them as a projection matrix1. PCA gives you smaller vectors, but also additional operational complexity. You need to store and version the PCA transformation, then apply the same version consistently when adding to and querying the index.

How I ran the experiment

The idea behind the experiment was simple: I would shrink the embeddings with each method, run the benchmarks, and see how much retrieval quality suffers at each size.

I generated all embeddings through OpenRouter and evaluated retrieval on eight BEIR datasets: SciFact, NFCorpus, ArguAna, FiQA, SciDocs, Quora, TREC-COVID, and Webis-Touché 2020. For each dataset, I reduced the embeddings to 512, 256, 128, 64, and 32 dimensions.

NoteDataset details

dataset<br>task<br>corpus<br>queries

SciFact<br>scientific claim verification<br>5.2K<br>300

NFCorpus<br>medical search<br>3.6K<br>323

ArguAna<br>counterargument retrieval<br>8.7K<br>1,406

FiQA-2018<br>financial question answering<br>57K<br>648

SciDocs<br>citation recommendation<br>25K<br>1,000

Quora<br>duplicate question retrieval<br>523K<br>10,000

TREC-COVID<br>biomedical search (COVID-19)<br>171K<br>50

Touché 2020<br>argument retrieval from web docs<br>382K<br>49

There were three questions I wanted to answer.

Q1: which method keeps more retrieval quality as you cut dims?

To answer this, I used two MRL-trained models: OpenAI’s text-embedding-3-small (1,536 dims), and Alibaba’s qwen3-embedding-8b (4,096 dims), which sits at the top of the open-weights MTEB BEIR leaderboard.

I reduced each model’s embeddings in two ways:

With truncation , I kept the first d dimensions and re-normalized. This is how you reduce dimensions with MRL.

With PCA , I fit the projection on the full-dimension document embeddings of the same dataset I then searched, and kept the top d components. From then on, every document and query embedding gets multiplied by that projection matrix and re-normalized before searching.

Then I ran the benchmarks on the eight datasets, comparing the two methods on both models, to see which one preserves more retrieval quality as the embeddings get smaller.

Q2: is it just the MRL training?

If PCA does well, that might be because MRL training has already organized the embedding space in a convenient way. To test that, I added text-embedding-ada-002, an older 1,536-dimensional model that wasn’t trained using MRL, as a control.

I applied PCA to both models. If PCA retains a similar share of retrieval quality on text-embedding-3-small and text-embedding-ada-002, that suggests its performance does not depend on MRL training.

Q3: does the fitting data matter?

PCA has to be fit on something, which raises two practical questions: how much fitting data do you need, and does it need to come from the corpus you’ll be searching?

For the first question, I fit PCA on random samples of FiQA’s 57K documents (1,000, 5,000, 20,000, and the full corpus) and checked how much the size of the fitting sample changes retrieval quality. This covers the scenario where you fit PCA on a sample of your data and never update it as the...