Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090 :: Giles' blog
el.dataset.currentDropdown = '')<br>}">
Giles' blog
Writing the post that I wished I'd found when I started learning whatever it was...
About
Contact
Archives
Categories
Blogroll
July 2026 (4)
June 2026 (7)
May 2026 (2)
April 2026 (11)
March 2026 (3)
February 2026 (4)
January 2026 (4)
December 2025 (1)
November 2025 (3)
October 2025 (9)
September 2025 (3)
August 2025 (5)
July 2025 (1)
June 2025 (2)
May 2025 (3)
April 2025 (2)
March 2025 (7)
February 2025 (10)
January 2025 (6)
December 2024 (7)
September 2024 (1)
August 2024 (2)
July 2024 (2)
May 2024 (2)
April 2024 (2)
February 2024 (2)
April 2023 (1)
March 2023 (2)
September 2022 (1)
February 2022 (1)
November 2021 (1)
March 2021 (1)
February 2021 (2)
August 2019 (1)
November 2018 (1)
May 2017 (1)
December 2016 (1)
April 2016 (1)
August 2015 (1)
December 2014 (1)
August 2014 (1)
March 2014 (1)
December 2013 (1)
October 2013 (3)
September 2013 (4)
August 2013 (2)
July 2013 (1)
June 2013 (1)
February 2013 (1)
October 2012 (1)
June 2012 (1)
May 2012 (1)
April 2012 (1)
February 2012 (1)
October 2011 (1)
June 2011 (1)
May 2011 (1)
April 2011 (1)
March 2011 (1)
February 2011 (1)
January 2011 (1)
December 2010 (3)
November 2010 (1)
October 2010 (1)
September 2010 (1)
August 2010 (1)
July 2010 (1)
May 2010 (3)
April 2010 (1)
March 2010 (2)
February 2010 (3)
January 2010 (4)
December 2009 (2)
November 2009 (5)
October 2009 (2)
September 2009 (2)
August 2009 (3)
July 2009 (1)
May 2009 (1)
April 2009 (1)
March 2009 (5)
February 2009 (5)
January 2009 (5)
December 2008 (3)
November 2008 (7)
October 2008 (4)
September 2008 (2)
August 2008 (1)
July 2008 (1)
June 2008 (1)
May 2008 (1)
April 2008 (1)
January 2008 (4)
December 2007 (3)
March 2007 (3)
February 2007 (1)
January 2007 (2)
December 2006 (4)
November 2006 (18)
AI (91)
TIL deep dives (77)
Python (72)
LLM from scratch (48)
Resolver One (34)
PyTorch (21)
TIL (21)
Blogkeeping (18)
PythonAnywhere (17)
Linux (16)
Startups (15)
Gadgets (13)
Hugging Face (13)
NSLU2 offsite backup project (13)
Funny (11)
Musings (11)
Finance (10)
Fine-tuning LLMs (10)
C (9)
Personal (8)
Robotics (8)
Website design (8)
JAX (7)
3D (5)
Quick links (5)
Rants (5)
Cryptography (4)
JavaScript (4)
Music (4)
Oddities (4)
Talks (4)
Dirigible (3)
Eee (3)
Memes (3)
Politics (3)
Django (2)
GPU Computing (2)
LaTeX (2)
MathML (2)
Microprojects (2)
OLPC XO (2)
Retro Language Models (2)
Space (2)
VoIP (2)
Copyright (1)
Golang (1)
poppy the training box (1)
Raspberry Pi (1)
Software development tools (1)
Agile Abstractions
Astral Codex Ten
:: (Bloggable a) => a -> IO ()
David Friedman's Substack
Econ & Energy
Entrepreneurial Geekiness
For some value of "Magic"
Hackaday
kaleidic.ai newsletter
Knowing.NET
Language Log
Millennium Hand
ntoll.org
Obey the Testing Goat!
PK
PythonAnywhere News
Simon Willison's Weblog
Societive
Software Deviser
Some opinions, held with varying degrees of certainty
tartley.com
Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090
Posted on 24 July 2026
in
Microprojects,
AI
I mentioned I'd got a second RTX 3090 on a group chat, and a friend said:
I know this is not really your thing... but let me know how quickly it runs<br>Qwen 3.6 35bn MoE. With only 24gb of VRAM you’ll need to use a 4-bit quantized<br>version and you won’t get a massive context window. But it should still be<br>pretty cool.
He's right that it's not really been my thing -- I've been focusing on my<br>own LLMs recently. I decided to dig in a little, and in particular to play with<br>Llama.cpp, which I haven't used for a while. And then things got<br>a tad out of control, and I wound up doing some relatively detailed benchmarking.
The headline results: I downloaded Unsloth's UD-IQ4_NL_XL quantisation of the model<br>from Hugging Face. With that,<br>using the default Arch build of Llama.cpp, which uses Vulkan under the hood:
Using the GPU only, I was able to get the model to generate at just over 120 tokens per second,<br>and it was able to process the prompt at just less than 2,800 tok/s. However,<br>having the whole model on the GPU didn't leave that much space for the context<br>window -- it was constrained to about 50,000 tokens, compared to the model's<br>native context length of 262,144.
Offloading the FFNs for the first 12 of the model's 40 layers to the CPU managed to reclaim<br>enough VRAM to be able to get the full context length; however, with that setup,<br>things were -- unsurprisingly -- slower. I got just over 65 tok/s for generation,<br>and 600 tok/s for the prompt.
Compiling Llama.cpp myself, in order to get the full CUDA version, helped a lot:
With everything on the GPU, I got 140 tok/s for generation and over 3,300 tok/s<br>for the prompt. That was with a context window of 89,600. So everything was<br>better :-)
It was also easier to get...