Local vs Frontier: Benchmarks for my Digital Nervous System – Human Paradox - Colin's Blog

Local vs Frontier: Benchmarks for my Digital Nervous System

02 Aug, 2026

LLM benchmarks are a contentious, if often unserious topic. There are many benchmarks. Some are made by companies that make LLMs. Some are hilariously pie-in-the-sky like AGI ones. But most of them are just a number, and most of us just look for "does number go up?" On top of that, you'll see all over the internet: "I use and it's just as good as ." But you rarely see, in these type of comments, any actual details about what they are trying to do with the models.

As I've been building out my "Digital Nervous System", I decided to test out which local models would potentially be best, and what compromises I would be making. If you download any local model and ask "What do you know about " they will all pretty much regurgitate Wikipedia, so that's a pretty useless personal benchmark. On top of that, while my digital nervous system tasks are not technically challenging, they are quite nuanced. Things like: I work remotely, so my work events natively are in a different time zone and my calendars often render UTC offsets rather than store them. Or some do both. Or I'm remote but everything has a room in the office on the invite. Or I have the Yankees schedule as a calendar but that doesn't mean I'm going to every game, etc.

To test the current state of local and frontier models in my digital nervous system context, I set up four benchmarks:

Figure out who my sister is.

Figure out who had a higher GPA in college - me, or my wife?

Figure out why there are 4 different dentist appointments on my calendar last Thursday.

Look at the MotionCor3 source code and figure out whether TIFF decompression is faster if done on the CPU or the GPU.

What kind of benchmark questions are these? All a bit deeper than they look. Here's the explanations for each:

I've developed an internal tool called K_ir. This is basically an annotated, embedded database with lexical and semantic search over every digital record I control or possess. This includes email, calendar, messaging apps, etc. This question is tricky because it needs to understand who I am, figure out who my sister might be by name search, but also incorporate that her name has changed post-marriage. This is the type of thinking that doesn't always show in a benchmark.

Similar to the above, it must understand who I am, as well as who my wife is, and figure out her name change. Also, there are no official GPA documents sitting around anywhere, so it will need to be decent at searching and sorting. This test, as well as the above, is meant to represent how useful the models are with incomplete instructions that rely on context and inference to follow through - the exact type of use case a personal assistant will need heavily.

Direct result of previous attempts at a personal assistant, and calendar timezone hijinks, but also with a gotcha built in. I somehow wound up with dentist appointments listed in my calendar as a result of timezone malfunctions, however, as a result of rendering malfunctions, there's also a fourth showing at a different time. A tricky question to deconvolute.

OK this is at first a bog standard coding question. MotionCor3 is used in cryo-EM to align raw, non-motion corrected frames to a single motion corrected frame. These are often large (500 MB+) TIFF files that have been LZW compressed, and must be decompressed before the GPU can do the frame alignment. Obviously, GPUs tend to be much faster 1:1 than CPUs at decompression, plus you are sending a smaller file through the PCI-E bus, so you'd suspect that GPU would win. However, if you are spending GPU cycles on decompression, they are not available for the motion correction. Therefore, it's faster overall to do decompression on CPU. This isn't really ever explicitly stated in the codebase, but there is some forensic evidence that points to it.

I tested the following models:

Qwen 3.6 35B-A3B

Qwen 3.6 27B

Bonsai 27B

DeepSeek V4 Flash (API)

Laguna 2.1S (API)

Claude Sonnet 5 (Subscription) (Thinking: High)

Claude Opus 5 (Subscription) (Thinking: High)

Results (No Claude)<br>1) Who is my sister?<br>All five non-Claude models got the right person, but they went about it differently, and some clearly did it worse.

Qwen 27B and V4 Flash did it properly: first, they figured out who I was, looked at the entity graph in K_ir, found my asserted relations, noted that the maiden name is an alias on the same record, and produced the result. V4 Flash noticed that a plain lexical search for "sister" is dominated by a band called Sister John Angela (highly recommend btw), and determined that the entity graph was the right tool because the relation was asserted rather than inferred. Laguna started with lexical search but eventually got there via the assertion.

Qwen 35B-A3B got the right name without ever querying K_ir. It searched the agent's...