GitHub - mbuchel/sass2mlir-bench: this is benchmarking for sass2mlir · GitHub

/" data-turbo-transient="true" />

Skip to content

Type / to search

Sign in<br>Sign upAppearance settings

You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

{{ message }}

mbuchel

sass2mlir-bench

Public

Notifications<br>You must be signed in to change notification settings

Fork

Star

master

BranchesTags

Go to file

CodeOpen more actions menu

Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit

History<br>2 Commits<br>2 Commits

passes/example

passes/example

.gitignore

.gitignore

LICENSE

LICENSE

Makefile

Makefile

PASSES.md

PASSES.md

README.md

README.md

bench.sh

bench.sh

bench_vllm.py

bench_vllm.py

setup.sh

setup.sh

speedups.jpg

speedups.jpg

View all files

Repository files navigation

jetson-llm-bench

Stock LLM decode benchmarks for the Jetson AGX Orin (and an x86_64<br>desktop): llama.cpp on GGUF, vLLM on bf16 safetensors. No custom kernels,<br>no interposers — this is the reference harness you quote baselines with.

For the technical explanation of what the passes do and how they get<br>these numbers, see<br>mbuchel.github.io/projects/sass2mlir .

Two hosts, detected from uname -m; everything that differs follows<br>from that:

Jetson AGX Orin<br>Blackwell desktop

arch<br>aarch64, sm_87, 14 SM<br>x86_64, sm_120

engines<br>llama.cpp + vLLM<br>llama.cpp + vLLM

wheels<br>jetson-ai-lab index, pinned<br>PyPI (its x86_64 wheels are CUDA builds)

Override any of it: ARCH=, CUDA_ARCH=, INDEX_URL=, VLLM_SPEC=.

Measured speedups

Stock llama.cpp against the same model under a pass built with this dev<br>flow — Jetson AGX Orin, tg128, stock and pass measured back to back in<br>one session:

Qwen3.5-4B (Q4_K_M): 25.0 → 31.0 tok/s

Nemotron3-Nano-4B (Q4_K_M): 31.2 → 40.5 tok/s

The venv is made by uv — ./setup.sh vllm<br>installs it into ~/.local/bin itself if it is missing (rc files<br>untouched), because its resolver is much faster than pip on eMMC. If uv<br>cannot be installed, it falls back to python3 -m venv + pip — same<br>.venv layout either way. The interpreter stays 3.10 regardless: the<br>jetson-ai-lab wheels are cp310 builds.

./setup.sh # llama-bench + vLLM venv + GGUFs<br>./setup.sh models # just the GGUFs (~5 GB)<br>./setup.sh bf16 # + Qwen bf16 safetensors (~8 GB, for vLLM)<br>./setup.sh check # what is present, what is missing<br>sudo jetson_clocks # before any measurement

MODEL_GGUF=$PWD/models/Qwen3.5-4B-Q4_K_M.gguf ./bench.sh llama<br>MODEL_HF=$PWD/models/Qwen3.5-4B ./bench.sh vllm<br>MODEL_HF=$PWD/models/Qwen3.5-4B ./bench.sh vllm-eager

Model formats: why there are two downloads

llama.cpp reads GGUF and nothing else. It has no safetensors reader,<br>so the llama.cpp side of each model exists only as a GGUF — for<br>Qwen3.5 that is Qwen3.5-4B-Q4_K_M.gguf (~4.8 bpw), and that quantized<br>file is the llama baseline, not an optional extra.

vLLM serves the bf16 safetensors (Qwen/Qwen3.5-4B, ~8 GB), fetched<br>separately by ./setup.sh bf16 because only the vLLM side consumes them.

Consequence for reading numbers: the llama.cpp run is Q4_K_M and the<br>vLLM run is bf16 — the two engines are not precision-matched , and<br>llama.cpp moves ~3x fewer weight bytes per token. Their tok/s are not<br>directly comparable; quote each engine against itself.

Models

repo<br>file<br>used by

Nemotron GGUF<br>nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF<br>NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf<br>llama.cpp

Qwen GGUF<br>unsloth/Qwen3.5-4B-GGUF<br>Qwen3.5-4B-Q4_K_M.gguf<br>llama.cpp

Qwen bf16<br>Qwen/Qwen3.5-4B<br>models/Qwen3.5-4B/ (dir)<br>vLLM

Downloads resume, so a dropped link costs nothing. On x86_64 the Qwen<br>GGUF is skipped by default to keep the download small; QWEN=1 ./setup.sh models fetches it anyway.

Writing a pass

The .deb doubles as the dev package: it installs the headers and static<br>libraries a pass links against, and this repo builds one out of tree, so<br>iterating on the Jetson does not mean rebuilding sass2mlir (which itself<br>always comes from the .deb — nothing here builds it). See<br>PASSES.md .

cp -r passes/example passes/my-pass # then edit<br>make my-pass<br>sass2mlir my-pass some.cubin --passes-dir passes/my-pass

Why --index-url and not --extra-index-url

The jetson-ai-lab index is where the CUDA-enabled aarch64 wheels live —<br>vllm 0.20.0+cu126, torch 2.11.0, triton, flash-attn, all built<br>for JetPack 6 / CUDA 12.6. That is the whole point of pointing at it,<br>and it means torch needs no special case: it arrives as a vLLM<br>dependency from the same index.

--extra-index-url would add PyPI alongside it. PyPI carries generic<br>aarch64 builds under the same names, and pip picks by version, not by<br>which index looked more authoritative — so a higher version number on<br>PyPI wins and you get a wheel with no CUDA. That is not an install<br>error: it is a working import with no GPU, found much later and<br>confusingly. --index-url...