Client-side semantic search for your static site | Bart de GoedeListen to this article instead
Your browser does not support the audio element<br>Eight years ago I added client-side search to this blog with Lunr.js. It creates an inverted index at build time, ships it as JSON, and matches strings in your browser. No server-side engine required. It has worked fine ever since, in the sense that it finds a post if you type a word that is actually in it.<br>Earlier this year I wrote a semantic search engine in ±250 lines of Python (the kind that lets you find the “London Beer Flood” when you search for “alcoholic beverage disaster in England,” because it understands that beer is alcoholic and a flood is a disaster). That one needs a machine with sentence-transformers installed and a few hundred megabytes of PyTorch; to serve something like that in a production server requires beefy machines with expensive RAM. Not something you run in a browser tab.<br>So this blog had keyword search that runs anywhere and understands nothing, but no semantic search that understands things but can’t run anywhere near a static site. This post is about closing that gap: semantic search that runs entirely in your browser, with no server and no API, where the entire model is essentially a 4 MB lookup table. You can try all three of the models I benchmarked, further down, running live on your own hardware (I can’t afford GPU clusters).<br>Doing this surfaced a couple things, chief among which that the keyword search had a bug for a couple of years. TUrns out that doing proper evals is important.<br>The obvious approach costs 23 megabytes#<br>The models that power the Python post (sentence-transformers like all-MiniLM-L6-v2) can, in fact, run in a browser. Transformers.js will download an ONNX build of one and run it on WebAssembly. I benchmarked it, and on my laptop, the quantized model plus its runtime is 23.45 MB over the wire, takes about two seconds to load and become available, and then embeds a query in ±18 ms.<br>Twenty-three megabytes is a bit rich to embed a search box query, for a blog that has 14 posts. That is roughly a dozen high-resolution photos worth of bytes to download so we can have fancy search. It works, and for some applications it is completely worth it, but it is not something I want to inflict on someone who clicked through to read about bloom filters, especially not on a mobile connection.<br>The thing is, you do not need a transformer to embed a search query. You need a vector that’s good enough, and there is a much cheaper way to get one.<br>A static embedding model is a lookup table#<br>The trick is a family of models called model2vec (the specific ones are named “potion”). They are distilled from a real sentence-transformer, but the result is not a neural network. It is a table.<br>Here is the model’s entire forward pass. Not a simplification — the whole thing, from the library’s source:<br>ids = tokenize(text) # split into subword token ids<br>rows = embedding[ids] # look up one vector per token<br>vector = rows.mean(axis=0) # average them<br>vector = vector / norm(vector) # normalise to unit length
That is it. Tokenize, look up a row per token, average, normalize. There is no attention, no layers, no inference. “Running the model” is a handful of array lookups and an average. For potion-base-8M, the table is 29,528 tokens by 256 dimensions of float32 (about 30 MB) and it reaches 81% of MiniLM’s retrieval quality while being, definitionally, a dictionary lookup.<br>Thirty megabytes is still too much, but a table is a much friendlier thing to shrink than a transformer.<br>Building the index: chunking, and a cache that never earns its keep#<br>Embeddings are generated at build time. On my laptop, whenever I run hugo, a Python script walks through the posts, strips out the front matter and the code blocks, splits each post into overlapping 600-character-ish chunks, embeds every chunk, and writes the vectors to a file the browser downloads when a user clicks the search input box.<br>The chunking matters more than I expected, because of that averaging step. A static embedding is the mean of its token vectors, and if you average an entire 15,000-character post into one vector, you get something that points at “generic English prose about software” and not much else. The rare, distinctive words (terms like pydub or mmh3) get drowned out by the hundreds of ordinary words around them. Chopping the post into smaller chunks helps keep those signals sharp. I’ll come back to this, because it turns out to be the key to why some models beat others.<br>I also built an embedding cache, keyed on a hash of each chunk’s text, so that re-embedding only touches chunks that changed. This turned out to be a waste of time and tokens. Embedding every chunk of this blog is a few hundred lookups and an average; it takes about ten milliseconds. I built a cache to speed up an...