Bigger Wasn’t Better: Benchmarking Small Models for digiKam’s Natural Language Search | Srirupa’s Blog

GSoC 2026 • digiKam • Post 3: The Benchmark, and the Fine-Tuning Decision

At the end of my last post I promised a comparison: Qwen2.5 against TinyLlama on real digiKam queries. This is that post. It grew a third model along the way, and the result surprised me enough that I want to walk through it honestly, because the tidy expectation I started with turned out to be wrong.

If you’re just joining: in the first post I introduced the goal - bringing natural-language search to digiKam, so you can find photos by describing them in plain English instead of filling in an advanced-search form. In the second post I walked through actually wiring a local LLM into a desktop app, and the lesson that surprised me: the model was the small part, and the pipeline around it: the prompt, the parser, the dictionary that catches ambiguity, did most of the real work. This post picks up the thread I left there: is the model I chose actually the right one?

The question underneath all of this is a practical one. digiKam’s natural language search runs a local, quantized model on the user’s own machine, no cloud, no API, your photos and your queries never leave your computer. That constraint is the whole point of the feature, and it’s also what makes model choice hard. You can’t just reach for the biggest, best model; it has to load and run on an ordinary laptop, next to digiKam itself, fast enough that a search doesn’t feel broken. So the real question isn’t “which model is best,” it’s “which model is the right balance for this job.”

I’d been running Qwen2.5-1.5B this whole time because it felt right. This post is me actually checking.

What I measured, and how

I built a small benchmark harness. It lives in core/tests/llm/, it’s a standalone Python script, and it does three things for every query: measures latency , measures peak memory , and scores structured-output accuracy , whether the model produced the correct search constraints.

The one thing I cared about most was fidelity: the benchmark had to test the real pipeline, not a convenient approximation of it. So it uses the exact prompt digiKam sends, transcribed straight from SearchPromptBuilder, and it feeds the model the same way the C++ backend does, as a raw prompt with no chat-template wrapping. If the benchmark and the app disagreed on how they talked to the model, the numbers would be fiction.

The test set is about 40 hand-labelled queries. Each one pairs a plain-English request with the constraints it should produce: “photos from 2023 rated 5 stars” should give a date range and a rating. I scored at the level of the model’s raw intent, before the resolver’s later cleanup steps, because I wanted to measure the model, not the pipeline wrapped around it.

Which brings me to the first thing I got wrong.

The benchmark caught my own mistakes first

My first run scored Qwen at 66%. I almost believed it.

Then I read the failures, and most of them weren’t the model. They were me, in the labels. I’d written that “pictures tagged sunset” should produce tag with the operator contains; the model produced eq; and when I checked the actual code, digiKam’s tag matching ignores the operator entirely and looks the tag up by name. So the model was right, my expected answer was wrong, and my benchmark was confidently marking a correct output as a failure.

There were a handful like that. A caption operator I’d mislabelled. And a latency problem that turned out to be the harness, not the model: I was letting the model generate all the way to its token limit, when the real backend stops the moment it has a complete JSON object. The model had been producing a correct answer and then rambling on past it; the app already knew to stop reading, and my benchmark had forgotten to. It was the same “knowing when to shut up” issue from last post, except this time the mistake was mine, in the harness. Once I fixed it to stop at the first complete object the way the backend does, median latency dropped from about 15 seconds to under 2.

I’m telling you this because it’s the most important thing the benchmark did. Before it could measure the model, it measured my assumptions, and several of them were wrong. A benchmark that only ever confirms what you expected isn’t measuring anything. The 66% was noise; the real signal was underneath, once I stopped trusting my own labels and started checking them against what the code actually does.

The honest Qwen2.5-1.5B number, after fixing my labels, is about 85% .

The three-way comparison

I benchmarked three models, all as Q4_K_M quantized GGUFs so the comparison is fair, all getting the identical prompt:

TinyLlama-1.1B , the lightweight baseline.

Qwen2.5-1.5B , the model I’d been using.

Qwen2.5-3B , added because I wanted to know: would a bigger model be better?

Here’s what came back:

Model<br>Constraint accuracy<br>Median latency<br>Peak...