Do Qwen 3.6 27B quantizations break the pelican? - Quesma BlogHow does model quantization impact its quality?
Could you guess which quantization was used? The largest 8-bit, the smallest 2-bit, or something in between?
Not long ago, I was raving about Qwen 3.6 27B. The Hacker News discussion was about whether you need a beast or can use a much smaller machine.
Even an 8-bit model required less than 45 GB of RAM - a lot for an Nvidia card (RTX 5090 is 32 GB RAM), but doable for a strong Apple Silicon laptop.
But with a smaller quantization we can go much smaller - less than 30 GB for Q4_K_M, and less than 18 GB for the smallest 2-bit quantization, UD-IQ2_XXS.
I was curious how quantization affects the result.<br>We will see how it affects results: by typical probability measurements, looking at SVG with pelicans and gears, and running benchmarks. Burned 37h on my own laptop (generating 3.9M tokens) and $1,430 on GPUs in the cloud - so you don’t have to.
Quantizations
Model quantization is a lossy compression. If you like, you can do it in a few minutes with llama.cpp,
llama-quantize Qwen3.6-27B-BF16-00001-of-00002.gguf \ # input weights<br>Qwen3.6-27B-Q4_K_M-llama.gguf \ # output file<br>Q4_K_M # quantization<br>But usually you get better results when it is done with care - downloading one from Hugging Face by Bartowski or Unsloth.
All GGUF quantizations of Qwen 3.6 27B by Unsloth, from 9.6 GB (UD-IQ2_XXS) to 54.7 GB (BF16).
The names split into a few families, which are the colors in every chart below (IQ and legacy quants share one):
standard K-quants (Q4_K_M, Q6_K) are the llama.cpp default. They store weights in small blocks, with a shared scale per block. The _S/_M/_L suffix tells how many tensors stay at higher precision.
IQ quants (IQ4_XS, IQ4_NL) compress with a codebook and an importance matrix.
legacy quants (Q4_0, Q4_1) are the oldest formats. They just round to the nearest value.
Unsloth Dynamic (UD) (UD-Q4_K_XL) checks which tensors are the most sensitive to compression, and gives them more bits.
Statistical measures
Quantization is a lossy compression. Models predict tokens, so we can measure if there is a different probability distribution.
Kullback-Leibler divergence
One statistical measure is Kullback-Leibler divergence, known also as relative entropy. In short, it is how unexpected results from the new distribution are with respect to the original distribution.
Download as PNG
0.0010.010.101020304050 GBmodel size on diskmean KL divergence from BF16UD-IQ2_XXSUD-IQ2_MUD-Q2_K_XLUD-IQ3_XXSQ3_K_SQ3_K_MUD-Q3_K_XLIQ4_XSQ4_0Q4_K_SIQ4_NLQ4_1UD-Q4_K_XLQ5_K_SQ5_K_MUD-Q5_K_XLQ6_KUD-Q6_K_XLQ8_0UD-Q8_K_XL<br>Unsloth Dynamic (UD)standard K-quantsIQ4 & legacy Q4BF16 reference
Here the reference BF16 is not shown, as it would be zero, which does not work on a logarithmic scale.<br>However, this chart does not answer one core question - does the difference actually matter.
Top-1
A simpler one: how often the quantized model picks the same most likely token as the original.<br>This is what matters at temperature zero, where only the top token gets used.
Download as PNG
80%85%90%95%100%01020304050 GBmodel size on disksame top-1 token as BF16UD-IQ2_XXSUD-IQ2_MUD-Q2_K_XLUD-IQ3_XXSQ3_K_SQ3_K_MUD-Q3_K_XLIQ4_XSQ4_0Q4_K_SIQ4_NLQ4_1UD-Q4_K_XLQ5_K_SQ5_K_MQ6_KUD-Q6_K_XLQ8_0UD-Q8_K_XLBF16<br>Unsloth Dynamic (UD)standard K-quantsIQ4 & legacy Q4BF16 reference
Here we see that Q8_0 is 99.3%, so almost the same as the original. Then things fall down with models getting smaller.
Collision cross-entropy
However, if we work at temperature 1, we need to look at all tokens.<br>Even running the same model twice gives different results. We can measure the chance of getting the same token as the reference.
Download as PNG
44%46%48%01020304050 GBmodel size on disksame sampled token as BF16, temp 1UD-IQ2_XXSUD-IQ2_MUD-Q2_K_XLUD-IQ3_XXSQ3_K_SQ3_K_MUD-Q3_K_XLQ4_0Q4_1Q5_K_SUD-Q5_K_XLQ6_KQ8_0UD-Q8_K_XLBF16<br>Unsloth Dynamic (UD)standard K-quantsIQ4 & legacy Q4BF16 reference
For the same, it is 48.5%. For others, we see that this range stays largely the same, until there is a sharp drop around 3-bit quantization.
Perplexity
Another measure is how well we can predict the next token of a text, perplexity, e^(log-loss): roughly, between how many equally likely tokens the model picks at each step. Putting it here for the sake of completeness.
Download as PNG
6.577.5801020304050 GBmodel size on diskperplexity on wikitext-2UD-IQ2_XXSUD-IQ2_MUD-Q2_K_XLUD-IQ3_XXSQ3_K_SQ3_K_MUD-Q3_K_XLIQ4_XSQ4_0Q5_K_SUD-Q5_K_XLQ6_KUD-Q6_K_XLQ8_0UD-Q8_K_XLBF16<br>Unsloth Dynamic (UD)standard K-quantsIQ4 & legacy Q4BF16 reference
Looking at SVGs
But, does it matter?<br>I decided to compare models, with temperature 0 and one-shots to generate SVGs.<br>I expected that at some point quality will deteriorate, or maybe even it won’t be able to create a working SVG at all.
Pelican
The most classical “Hello World” is Simon Willison’s pelican. So, here we go:
Generate an...