Latent Reasoning
Command Palette<br>Search for a command to run...
NNicholai Mitchko
Personal Research - Scaling AI Use @ InterSystems
Comments<br>Join the discussion<br>No comments yet. Be the first to comment.
More from this blog<br>Jul 17, 2026·12 min read·169
Mar 31, 2026·5 min read·211
AI Research<br>3 posts
div>p:first-child]:mt-0 [&>div>p:first-child]:pt-0 min-w-0 wrap-break-word [&_a]:break-all **:max-w-full">DeepSeek-V4-Flash-0731-Latent-Reasoning. A self-contained model that does thinking in latent space, NVFP4-quantized, with a production vllm form for serving runtime.
Published on blog.n.ichol.ai
Where the last edition left off
Last time I grafted a CoLaR head (Compressed Latent Reasoning) onto DeepSeek Flash v4. The work had all the makings of a demo. A head that lets the model think in latent space. A learned stop head that decides when it has thought enough. A riddle that showed why plain autoregressive generation regurgitates cached answers instead of reasoning.
There was one honest problem though: the head was an adapter. Something bolted on the side. To serve it you had to assemble the base model, the head, the stop criterion, and a custom runtime by hand, then hope the pieces fit. The weights lived in one place. The inference machinery in another. "Here's how you run it" was a jump-through-hoops story.
This edition closes that gap. The work is now a complete, self-contained model . Every weight needed to serve it ships in one repository. The backbone is quantized down to NVFP4 so it fits on real silicon. And the latent loop is driven by a proper, benchmarked serving runtime.
Model: nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning
The big change: this is not an adapter anymore
The whole point of this edition is packaging. The old release was a head you had to attach. The new one is a model.
Every weight needed to serve it now sits in one HuggingFace repo:
The DeepSeek-V4-Flash-0731 backbone , quantized to NVFP4 (group size 16) on the routed MoE experts. Attention, shared experts, LM head and draft block stay at higher precision. Roughly 79 GiB of weights per GPU at TP=2 (158–164 GiB total).
The DSpark draft block (3 layers), preserved from the source, so speculative decoding ships in the box.
The trained latent reasoning head . 35.7M params, loaded from a single latent_reasoning_head.safetensors (~152 MB).
Because the weights are complete, the model card can finally report a real benchmark instead of "more benchmarks soon."
Actually writing down the numbers this time
The previous post ended with a half-promise: "full benchmarking to come if I find time." I found time.
BBH (BIG-Bench Hard), cot_zeroshot, 27 subtasks: aggregate 0.880 ± 0.008. Measured with lm-evaluation-harness 0.4.12 against an OpenAI-compatible endpoint. Thinking enabled. 50 items per subtask, 1350 items total.
Subtask<br>Score<br>Subtask<br>Score
tracking_shuffled_objects_three_objects<br>1.00<br>date_understanding<br>0.92
tracking_shuffled_objects_five_objects<br>1.00<br>sports_understanding<br>0.88
tracking_shuffled_objects_seven_objects<br>1.00<br>logical_deduction_five_objects<br>0.88
penguins_in_a_table<br>1.00<br>web_of_lies<br>0.86
formal_fallacies<br>1.00<br>snarks<br>0.84
boolean_expressions<br>1.00<br>ruin_names<br>0.84
word_sorting<br>0.98<br>movie_recommendation<br>0.84
temporal_sequences<br>0.98<br>salient_translation_error_detection<br>0.76
object_counting<br>0.98<br>geometric_shapes<br>0.74
navigate<br>0.98<br>causal_judgement<br>0.66
logical_deduction_three_objects<br>0.98<br>disambiguation_qa<br>0.58
reasoning_about_colored_objects<br>0.96<br>dyck_languages<br>0.26
hyperbaton<br>0.96
multistep_arithmetic_two<br>0.94
logical_deduction_seven_objects<br>0.94
The pattern is exactly what you would hope for from a latent reasoning model. It is strongest where reasoning means multi-step state tracking. tracking_shuffled_objects, boolean_expressions, formal_fallacies, penguins_in_a_table all hit 1.00. It is weakest on the mechanical, syntax-heavy jobs. dyck_languages (bracket matching) sits at 0.26, the clear outlier. That is a genuine weakness, not a measurement artifact.
Two honest notes on reading the table:
Read flexible-extract, not strict-match. BBH's strict-match regexes for the literal phrase The answer is X. This model does not emit that phrase, because it reasons in latent space. Its near-zero strict-match score is an answer-formatting artifact, not a reasoning failure.
Per-subtask values carry about ±0.05–0.07 at 50 items each. The aggregate of 0.880 is the reliable number.
Why the head looks different now
The architecture picks up where the original CoLaR idea left off, but it has a proper home in the model now. A small head reads the backbone's layer-35 hidden state, projects it into a 1024-d latent , and decodes it back into the residual stream. One latent step stands in for several reasoning tokens (a recorded compression_factor of 6). A learned stop head self-terminates the loop at a variable, content-dependent depth.
layer 35 hidden (4096-d)<br>v LayerNorm<br>+---------...