LLM Inference: VRAM & Performance Calculator

🧮'>

LLM Inference: VRAM & Performance Calculator

Pick a model, a quantization and a device — see whether it fits and how fast it runs.

Model

Inference quantization<br>Precision of the model weights. Lower uses less VRAM but costs quality.

KV cache quantization<br>KV cache precision. Dominates VRAM at long context.

Hardware

Device / GPU<br>Select your GPU or configure a custom device.

VRAM (GB)

Bandwidth (GB/s)

FP16 TFLOPS

Number of devices1

Tensor-parallel replicas. Comms overhead is included.

12481632

Enable offloading to CPU RAM / NVMe<br>Lets the model exceed VRAM — at host-bandwidth speed.

Workload

Batch size1

Sequences processed per step. Raises throughput, costs KV cache.

141664128

Sequence length1,024

Tokens per sequence (prompt + generation). Drives KV cache.

Concurrent users1

Simultaneous requests. Multiplies KV cache, splits per-user speed.

141664128

0% of VRAM

Comfortable<br>0 GB

of 0 GB usable

Performance<br>Energy & cost

Generation speed–

Per-token latency–

Time to first token–

Total throughput–

Bottleneck–

Memory allocation

Power draw–

Cloud rental (on-demand)–

Cloud rental (spot)–

Monthly rental, 24/7–

Electricity & grid

Electricity / day–

Electricity / month–

Electricity / year–

CO₂ / year–

Assumes the device runs continuously at the estimated utilization.