pdd/research/omlx-qwen38-quantization at main · promptdriven/pdd · GitHub
//files/disambiguate" data-turbo-transient="true" />
Skip to content
Search/
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Uh oh!
There was an error while loading. Please reload this page.
promptdriven
pdd
Public
Notifications<br>You must be signed in to change notification settings
Fork<br>74
Star<br>842
FilesExpand file tree
main
/omlx-qwen38-quantization<br>Copy path
Directory actions
More options<br>More options
Directory actions
More options<br>More options
Latest commit
History<br>History<br>History
main
/omlx-qwen38-quantization<br>Copy path
Top
Folders and files<br>NameNameLast commit message<br>Last commit date<br>parent directory<br>..<br>results
results
README.md
README.md
analyze_results.py
analyze_results.py
artifact-sha256.txt
artifact-sha256.txt
benchmark.py
benchmark.py
View all files
README.md<br>Outline<br>Qwen3.8 27B BF16 vs non-strict oQ8e on Apple M4 Max
This directory records a descriptive, single-session local observation of<br>Qwen3.8-27B BF16+MTP versus a locally converted 8-bit affine group-64<br>oQ8e+MTP checkpoint in oMLX 0.6.1.
The strongest defensible result is narrow: the oQ8e checkpoint was about 46%<br>smaller on disk, used substantially less observed process memory, and decoded<br>about 1.5–1.7x faster in this session. It was not faster at every workload :<br>16K prefill, time to first token, and end-to-end latency were worse. The coding<br>sample had the same pass/fail result for both models, but it is small and partly<br>test-exposed, so it does not establish general accuracy parity.
Environment
MacBook Pro, Apple M4 Max, 16 CPU cores, 128 GB unified memory
macOS 26.5.2
oMLX 0.6.1
Source checkpoint: fcmeyer/Qwen3.8-27B-MLX-bf16-mtp
Pinned source revision: fe34c8d6784c6d9b463756dd020492123137b732
Native Lightning MTP enabled; TurboQuant KV, DFlash, SpecPrefill, VLM MTP,<br>thinking, and remote code disabled
No machine serial number, hardware UUID, API key, or unredacted home-directory<br>path is included in this repository.
Artifact identity
The locally converted checkpoint used:
"oq_level": 8,<br>"group_size": 64,<br>"dtype": "bfloat16",<br>"preserve_mtp": true,<br>"text_only": false,<br>"auto_proxy_sensitivity": true,<br>"enhanced": true,<br>"imatrix_reuse_cache": true,<br>"imatrix_num_samples": 128,<br>"imatrix_seq_length": 512,<br>"imatrix_strict": false
Strict conversion first aborted because the imatrix lacked<br>language_model.model.embed_tokens. The completed artifact is therefore<br>non-strict oQ8e . Its report records 503 imatrix applications, with<br>language_model.model.embed_tokens and the tied language_model.lm_head<br>falling back to standard oQ8. The MTP projections are quantized, while<br>language_model.mtp.fc.weight remains BF16. “MTP enabled” here means the native<br>MTP path was active and logged draft acceptance/cycles; it does not imply that<br>BF16 and oQ8e produce bit-identical outputs.
The output contained six safetensors shards, 2,209 indexed tensors, exact<br>non-overlapping offsets, no trailing payload, and only regular non-executable<br>model files. Full source/config/cache/output hashes are in<br>artifact-sha256.txt. Those hashes identify this local<br>conversion; they do not claim that a public model repository contains the same<br>derived files.
Throughput protocol
Built-in oMLX code_python benchmark through the local OpenAI-compatible<br>HTTP endpoint
Prompt targets: 1,024, 4,096, and 16,384 tokens
Generation target: 128 tokens
Quick warmup, three repeats per run
Fixed order: BF16 pre-bracket, oQ8e, BF16 post-bracket
BF16 medians pool the three pre- and three post-bracket observations
oQ8e medians use its three observations
processing_tps is endpoint/TTFT-derived and is not raw engine-only prefill<br>throughput
Prompt target<br>BF16 decode median<br>oQ8e decode median<br>Decode ratio<br>BF16 processing median<br>oQ8e processing median<br>BF16 TTFT<br>oQ8e TTFT<br>BF16 E2E<br>oQ8e E2E
1K<br>8.95 tok/s<br>15.3 tok/s<br>1.71x<br>117.8 tok/s<br>93.1 tok/s<br>9.84 s<br>12.46 s<br>23.09 s<br>21.34 s
4K<br>10.15 tok/s<br>15.2 tok/s<br>1.50x<br>113.3 tok/s<br>113.7 tok/s<br>38.61 s<br>38.43 s<br>51.24 s<br>47.16 s
16K<br>8.85 tok/s<br>14.2 tok/s<br>1.60x<br>110.0 tok/s<br>95.6 tok/s<br>143.09 s<br>164.22 s<br>151.60 s<br>170.58 s
At 16K, oQ8e decoded faster but took about 12.5% longer end to end because<br>prefill/TTFT dominated. This is why the result should be described as a decode<br>speedup, not a universal latency speedup.
The BF16 pre-bracket was noisy: its 1K decode observations were<br>[20.3, 5.4, 10.0] tok/s. Bracketing exposes drift but does not remove thermal,<br>cache, power, or order effects.
Resource observations
Metric<br>BF16+MTP<br>oQ8e+MTP
Model files<br>51.75 GiB<br>27.97 GiB
Peak observed oMLX process footprint<br>57.78–58.08 GiB<br>37.53 GiB
Model load time<br>10.78–14.71 s<br>6.77 s
The footprint is an observed process-level...