Local LLM Benchmark: Administrative Tasks | AAI Labs

Loading…

Research paper2026<br>Local LLM Benchmark: Administrative Tasks<br>A DGX Spark evaluation of 13 locally run language models across a 21-task administrative work suite, covering calendaring, document parsing, financial and time-tracking calculations, email triage, and professional writing.<br>svg]:px-4 mt-8 bg-white text-black hover:bg-white/90">Read the paper

Abstract<br>This report evaluates 13 locally run language models on a benchmark of administrative work. The task suite covers calendar management, document parsing, financial and time-tracking calculations, email triage, and professional writing. All models were executed through Ollama on a single DGX Spark system with 128 GB of unified memory.<br>The evaluation combines deterministic Python checks for structured outputs with a local Llama 3.1 70B judge for open-ended writing tasks. This split makes it possible to compare models not only by aggregate score, but also by their suitability for different categories of administrative work.<br>Authors<br>Dmytro Klepachevskyi, Žygimantas Girdauskas, Robert Mackevič, Tadas Šubonis<br>What the study evaluates<br>The benchmark was designed around practical office tasks that require a model to read instructions, use tools or structured outputs, and produce work that can be checked. The 21 tasks include routine administrative operations as well as open-ended drafting tasks, allowing the study to distinguish between structured execution and prose quality.<br>Each model was run through the same agent harness and scored with the same criteria. Runs affected by GPU contention were repeated in isolation, so the reported results separate model behaviour from resource-scheduling artifacts.<br>Main findings<br>Gemma4:31b and Gemma4:12b achieved the strongest overall results and were statistically tied at the top of the benchmark. The 12B model is the more practical default for many deployments because it reaches nearly the same quality with a substantially smaller memory footprint.<br>Qwen3.6 was the strongest model for structured document work. It was the only model in the suite to pass the PDF-to-calendar task and led the non-Gemma models on automated structured tasks.<br>Model size was not a reliable predictor of performance. Several smaller models outperformed larger models once runtime conditions were controlled, which suggests that task fit and output behaviour matter more than parameter count alone for this class of administrative workflows.<br>Why it matters<br>Administrative automation depends on more than fluent text generation. A useful model must follow instructions over multiple steps, produce structured files or tool calls when required, and remain reliable under realistic hardware constraints. This benchmark makes those differences visible and gives practitioners a practical basis for model selection.