Which LLM can I run locally? VRAM and GPU calculator · Dubir Group

Free tool · No sign-upWhich LLM can I run locally?<br>Pick your hardware and what you want the model to do: chat, coding, reading PDFs and passports. The calculator lists the open models that fit, how much context they hold, roughly how fast they generate, and for everything that does not fit, why not.

Your hardware<br>AppleNVIDIAAMDIntelOther<br>GeForce<br>RTX 306012GB · 360 GB/sRTX 4060 Ti16GB · 288 GB/sRTX 5060 Ti16GB · 448 GB/sRTX 3090 (used)24GB · 936 GB/sRTX 409024GB · 1008 GB/sRTX 509032GB · 1792 GB/s2x RTX 309048GB · 936 GB/s

Workstation / data centre<br>RTX 6000 Ada48GB · 960 GB/sRTX PRO 600096GB · 1792 GB/sA100 (used)80GB · 2039 GB/sH10080GB · 3350 GB/s

GB10 desktops<br>DGX Spark128GB · 273 GB/sASUS GX10128GB · 273 GB/s

A dedicated GPU keeps about 90% for the model; the rest goes to context and activations.<br>What do you want it to do?<br>AnythingGeneral assistant / chatDocuments & scans (PDF, invoices, passports)Coding helpDrafting email & text24/7 pipeline (classify, extract, tag)Long documents & big context

Quantisation (compression)IQ2_M · ~2.8 bitsQ2_K · ~3 bitsQ3_K_M · ~3.9 bitsQ3_K_L · ~4.2 bitsIQ4_XS · ~4.3 bitsQ4_K_S · ~4.6 bitsQ4_K_M · ~4.85 bitsQ5_K_S · ~5.5 bitsQ5_K_M · ~5.7 bitsQ6_K · ~6.6 bitsQ8_0 · 8-bitFP16 · unquantised<br>The community default and still the sweet spot: about 0.6% off FP16 perplexity, which nobody notices in chat or summarisation. Strict formatting and long arithmetic degrade first.

Context length (working memory)2k tokens4k tokens8k tokens16k tokens32k tokens64k tokens128k tokens256k tokens<br>How much conversation or document the model holds at once. Every token costs KV-cache memory on top of the weights.

New to this? Quantisation and context, in a minuteA model ships as 16-bit numbers. Quantisation stores them in fewer bits, so the same model takes a half or a quarter of the memory and runs faster too, because generation speed is limited by how fast memory can be read. The cost is accuracy, and it is smaller than most people expect: at Q4_K_M, the default everywhere, the measured loss is under 1% and invisible in normal chat. Down at 3 bits it is a few percent and hard tasks get shaky; at 2 bits the model is visibly dumber. Above Q5 you cannot measure the difference without a test harness.<br>The rule that matters when shopping: a bigger model at 4-bit almost always beats a smaller one at 8-bit for the same memory.<br>Context length is the model's working memory: how much of the conversation or document it sees at once. It costs memory separately from the weights (the KV cache), which is why the table below sometimes says a model fits at 8k context but not at 128k. Runtimes such as LM Studio can also quantise the KV cache itself (turn on Flash Attention first): 8-bit cache halves that memory and costs essentially nothing.<br>Learn more: the loss figures are community measurements on real models, not ours. See Artefact2's KL-divergence tables, the llama.cpp k-quants work that defined these formats, and the LM Studio docs for the Flash Attention and KV-cache settings.

13 of 15 models run on RTX 4090 24GB at Q4_K_M with 8k of context. That device has 24 GB of memory, of which roughly 22 GB is yours to fill.

ModelGood atNeedsSpeedContext that fitsVerdictQwen3.6 35B-A3B (MoE)3B active parameters over a 35B pool, plus the same hybrid attention as the 27B.chat, 24/7 automation, long context20 GB~333 tok/s94k RunsQwen3-VL 30B-A3B (MoE) visionThe 30B-A3B recipe with eyes: 262k context swallows a whole contract as page images, and 3.3B active parameters keep it quick.documents & vision, chat, long context18 GB~302 tok/s44k RunsGemma 4 31B visionchat, coding, documents & vision19 GB~32 tok/s27k RunsQwen3 30B-A3B (MoE)A 2025 release that refuses to die: only 3.3B parameters run per token, so it is far faster than its size.chat, coding, 24/7 automation18 GB~302 tok/s40k RunsQwen3.6 27BHybrid attention: 48 of its 64 layers are linear-attention with a fixed-size state, so long context stays cheap.chat, coding, long context16 GB~37 tok/s102k RunsGemma 4 26B-A4B (MoE) visionchat, documents & vision, 24/7 automation15 GB~263 tok/s177k Runsgpt-oss 20B (MoE)Ships natively at ~4-bit (MXFP4); higher precisions are theoretical.chat, coding, 24/7 automation12 GB~277 tok/s128k RunsGemma 4 12B visionchat, documents & vision, drafting7.3 GB~83 tok/s237k RunsQwen3-VL 8B visionThe one-model answer to "read this invoice" and "describe this photo": DocVQA 96, OCRBench 90, and it fits an 8GB card at 4-bit.documents & vision, chat6.0 GB~115 tok/s119k RunsGemma 4 E4B visionchat, drafting, 24/7 automation4.6 GB~222 tok/s128k RunsGemma 4 E2B visionThe small end of Gemma 4. Also the standard draft model for speculative decoding in front of its bigger siblings.drafting, 24/7 automation2.9 GB~434 tok/s128k RunsDeepSeek-OCR-2 (MoE) visionMIT-licensed OCR built around optical compression: a dense page becomes a handful of visual tokens, which is exactly the...