The Model Was Never the Hard Part: Integrating Qwen2.5 into digiKam for Natural Language Search | Srirupa’s Blog

GSoC 2026 • digiKam • Post 2: Inference, Bugs, and the Build

In my first post, I introduced the goal: type a plain-English search into digiKam and have a local LLM translate it into structured search criteria. That post built the whole pipeline - prompt builder, JSON parser, intent resolver, against a mock backend that returned canned responses, so everything could be tested before a real model was wired in. This post is about swapping that mock for real llama.cpp inference, and everything that broke along the way, which was almost never the model.

At the end of my last post I promised that this one would be about the actual language model: which one, how fast, how accurate. I’ve been looking forward to writing it.

Here’s the thing I did not expect. The model works. It has essentially always worked. Almost every hard problem I hit over the past few weeks lived somewhere else: in a compiler flag, in a JSON type, in a git server’s opinions about submodules. This post is the honest version of what it takes to put a language model inside a desktop application, and the honest version is that the language model is the small part.

Actually running the thing

Last time the pipeline ran end-to-end against a mock backend: something that returned canned answers so I could build and test everything around it. Replacing that mock with a real model meant writing SearchLlamaBackend, which loads a quantized Qwen2.5 GGUF through llama.cpp and generates tokens.

Two decisions shaped it.

The first decision was that every single llama_* call happens on a worker thread . Loading a 1 GB model takes a few seconds; generating tokens takes a few more. If any of that ran on the GUI thread, digiKam would freeze every time you searched. So the backend owns a QThread, the worker lives on it, and everything crosses the boundary through queued signals - the UI stays responsive while the model thinks.

Here’s the shape of it (simplified from the real method, which has the error handling and tokenization removed for readability):

void SearchLlamaWorker::slotDoInference(const QString& prompt, int maxTokens, float temperature)<br>Q_UNUSED(temperature); // greedy decoding, determinism over creativity

llama_context* const ctx = static_castllama_context*>(m_context);<br>const llama_vocab* const vocab = llama_model_get_vocab(/* ... */);

// Start each query from an empty context.<br>llama_memory_clear(llama_get_memory(ctx), true);

// Greedy sampler: always pick the single most likely next token.<br>llama_sampler* smpl = llama_sampler_chain_init(llama_sampler_chain_default_params());<br>llama_sampler_chain_add(smpl, llama_sampler_init_greedy());

QString result;

while (generated maxTokens)<br>llama_decode(ctx, batch);<br>const llama_token tok = llama_sampler_sample(smpl, ctx, -1);

if (llama_vocab_is_eog(vocab, tok)) break;

result += /* decoded token text */;

// Stop as soon as the JSON object closes (balanced braces).<br>if (jsonObjectComplete(result)) break;

llama_sampler_free(smpl);<br>Q_EMIT signalOutputReady(result); // back to the main thread, via a queued signal

Two things in there are deliberate. llama_memory_clear at the top wipes the context’s KV cache so every query starts fresh - I’ll come back to why that one line matters more than it looks. And the sampler is greedy : no temperature, no randomness, the model always takes its single most likely token. That’s the opposite of how you’d run an LLM writing prose, where a little randomness keeps it from sounding wooden. But I don’t want prose. I want the same query to give the same JSON every time - so a bug is reproducible, and so the query cache from the last phase stores a real answer instead of one of several possible ones. For structured output, determinism isn’t a limitation; it’s the whole point.

Knowing when to shut up

A small problem I enjoyed solving. The model is supposed to emit one JSON object and stop. Sometimes it does. Sometimes it emits the object, decides it’s on a roll, and keeps going, producing helpful commentary, a second example, and whatever else it feels like until it hits the token limit.

Generating tokens you’re going to throw away is pure waste, and on a CPU each one costs real time. So the decode loop watches the output as it accumulates and counts brace depth. The moment the braces balance, meaning the first complete JSON object has closed, generation stops. In practice this cut a typical query from a hundred-plus tokens down to about twenty-two.

It’s a heuristic, and I know its failure mode: a } inside a string value would fool it. My schema doesn’t have string values that contain braces, so it holds. If that ever changes, the honest fix is to attempt a real parse each iteration and stop when it succeeds. I’d rather ship the simple thing that works and know exactly where it breaks.

Three bugs, none of them the model’s

Once real queries started flowing,...