Quantization hurts knowledge nonlinearly - Qwen3.6 27B case study - Quesma BlogIn previous blog posts, I was both raving about Qwen3.6 27B and investigating how its quantization affects pelican-making capabilities and benchmark scores. SVGs with pelicans on a bike were not affected by anything at 4 bits or more, and even for 3-bit and some 2-bit models the quality difference was small, visible only when averaged over many samples. Likewise, differences in benchmark results were surprisingly small.
So, why is it that everyday experience says smaller models are “worse”? While intelligence might not be that space-heavy, factual knowledge is incompressible - you cannot put the whole internet in a few GBs, no matter how hard you try.<br>Incompressible Knowledge Probes (IKP) is a 1,400-question trivia benchmark by Bojie Li. As we see, there is a strong linear correlation between the IKP accuracy and the logarithm of the number of parameters:
Download as PNG
0%20%40%60%80%100M1B10B100B1000Bmodel parameters (log scale)IKP accuracysmollm2-135mkimi-k2Qwen3.6 27B (BF16)<br>other modelsQwen3.6 27Bdashed: log-linear fit, +16 pp per 10× parameters
If you are curious about the questions spanning 7 tiers of obscurity, here are a few examples:
T1 : “What strait separates Sicily from mainland Italy?” - Strait of Messina
T2 : “What is the second largest moon of Saturn?” - Rhea
T3 : “Who composed the opera ‘Prince Igor’?” - Alexander Borodin
T4 : “What is the highest mountain in Oceania?” - Puncak Jaya
T5 : “In what year was Norwegian Olympic Museum founded?” - 1997
T6 : “In computer science, what is the research subfield of Myung Hoon Sunwoo?” - computer architecture
T7 : “In what year was Pont 1 (sot de les Mines) in El Brull, Spain opened?” - 1901
For reference, the score of the original model is the following:
T1T2T3T4T5T6T7Qwen3.6 27B99.5%97.5%78.5%38%10%3%4%
Let’s see how the score is affected by 55 quantizations: ones on Hugging Face by Unsloth and Bartowski, as well as ones by llama.cpp, with llama-quantize. We used llama.cpp to run these, locally on my MacBook M5, and on Modal. First, let’s decompose it by tier:
Download as PNG
0%25%50%75%100%1015203050 GBmodel size on disk, log scale (means over size bins)answered correctlyT1T2T3T4T5T6T7
So it all changes with model size, and the most obscure are lost first.
Let’s chart IKP accuracy as a function of the logarithm of model size.
Download as PNG
35%40%45%50%1015203050 GBmodel size on disk (log scale)IKP accuracyUD-IQ2_XXSIQ2_XXSUD-Q2_K_XLQ4_K_MBF16<br>UnslothBartowskillama.cppBF16 reference
Quantizations above 20 GB (5-bit or more) show the same quality as the original. Among 3-bit models there is a drop, which gets even stronger in the 2-bit range.
However, unlike in the main chart, the dependency is not linear. I had to double-check that it is not a problem with using different variables or scaling. So, let’s overlay it on the full chart.
Download as PNG
0%20%40%60%80%1101001000 GBmodel size on disk (log scale)IKP accuracyUD-IQ2_XXSQ4_K_MQwen3.6 27B (BF16)<br>UnslothBartowskillama.cppother models
Here it gets interesting. We get an arc over the chart. While Qwen3.6 27B already has a tagline “punches above its weight”, its slimmed-down 4-bit variants make this statement even… punchier (please excuse me, I just couldn’t resist).
Size matters, but divergence even more
This might look puzzling. Across models, IKP accuracy grows linearly with the logarithm of size. Within quantizations of a single model, the curve rises steeply, then flattens. When compressing a model (and quantization is a lossy compression ), what matters is not the size itself, but the statistical distribution of generated tokens. So, another chart might be more revealing, Kullback–Leibler divergence from the original, floating-point BF16 model:
Download as PNG
35%40%45%50%00.050.10.150.20.250.3mean KL divergence from BF16IKP accuracyr = -0.981UD-Q2_K_XLIQ3_SQ4_K_MQ8_0<br>UnslothBartowskillama.cpp
Here we have a beautiful linear correlation.
My interpretation is the following - when compressing a model a bit, we lose nothing, just getting rid of some needless noise. So we save space, at no expense.
But at some point the distribution starts breaking down, and quality falls more rapidly than it would with just training a smaller model. Because if we train a smaller model, we still optimize it for the training data. Quantization is blind. Even ones by Unsloth and Bartowski optimize for closeness to the original distribution, not fine-tuning data.
Consequences
This has two consequences. First, quantization is the way to go - you just need to know how far you can push it, so that there is no noticeable loss of quality. Usually you want to use the biggest model that fits your GPU RAM and leaves enough space for context. Unless you need your GPU RAM (or shared RAM, if it is Apple Silicon) for something else, there is no benefit in using cheaper models, usually there is no speed...