Running models on RISC-V with IREE - IREE

Skip to content

Initializing search

iree-org/iree

Guides

Deployment configurations

General topics

Reference

API bindings

MLIR dialects

MLIR extensions

Codegen/target-specific

Optional modules

IO

MLIR passes

Pipelines

Codegen

Developers

Building

Debugging

Performance

Design docs

Other topics

Community

Categories

Tags

Running the module

Benchmarking

Summary

Running models on RISC-V with IREElink

IREE compiles machine learning models to native RISC-V CPU code, with support for<br>the RISC-V Vector extension (RVV), hand-written microkernels, and data-tiling.<br>This post walks through the full flow for a model: importing it from PyTorch,<br>compiling it for a RISC-V target, running it, and benchmarking the result.

All commands below run under qemu-riscv64. The flow on real hardware is<br>identical — the QEMU invocation is simply replaced by running the tools natively<br>on the target.

Setuplink

IREE is a cross-compiler: the compiler is built on the host, the runtime is<br>cross-compiled for the target, and the runtime is then copied to the target (or<br>run under QEMU). The<br>RISC-V cross-compilation guide<br>covers this in full. In brief:

./build_tools/riscv/riscv_bootstrap.sh downloads a prebuilt clang toolchain<br>and QEMU into ~/riscv.

Build and install the host compiler, then cross-build the runtime with the<br>build_tools/cmake/linux_riscv64.cmake toolchain file.

Point QEMU_BIN at qemu-riscv64 and RISCV_TOOLCHAIN_ROOT at the toolchain.

The result is an iree-compile on the host and iree-run-module /<br>iree-benchmark-module built for RISC-V.

Importing a modellink

This post uses a few PyTorch models as a running example, but IREE supports<br>models from other frameworks such as LiteRT (TensorFlow Lite) and ONNX just as well,<br>once they have been imported to MLIR. See the<br>ML frameworks guides for the<br>per-framework export/import steps — for example<br>PyTorch,<br>LiteRT / TensorFlow Lite, and<br>ONNX. Also check out the<br>IREE community meeting<br>presentation by Artem<br>Gindinson from Roofline.

For PyTorch, iree-turbine's<br>aot.export produces the MLIR. The following script exports two torchvision<br>models, saving an input for each to feed later:

# export.py<br>import numpy as np, torch, torchvision as tv<br>import iree.turbine.aot as aot

def dump(name, model, example):<br>aot.export(model.eval(), example).save_mlir(f"{name}.mlir")<br>np.save(f"{name}_input.npy", example.numpy())

# Vision models (torchvision), NCHW float input.<br>dump("mobilenet", tv.models.mobilenet_v2(weights="DEFAULT"), torch.randn(1, 3, 224, 224))<br>dump("resnet18", tv.models.resnet18(weights="DEFAULT"), torch.randn(1, 3, 224, 224))

The exported entry point is @main, see the --function=main flag for the<br>iree-*-module invocations below.

Alternatively, you can check the models in the IREE test suites.<br>We have some ready-to-compile .mlir files whose weights are kept in a<br>separate .irpa<br>(IREE parameter archive) — which keeps the .mlir small and lets you swap weights<br>without recompiling. For example, Qwen3-0.6B:

curl -L -o qwen3.mlir https://raw.githubusercontent.com/iree-org/iree-test-suites/main/torch_models/qwen3-600m/model.mlir<br>curl -L -o qwen3.irpa https://huggingface.co/roofline/iree-regression-models/resolve/main/qwen3-600m/real_weights.irpa

The weights are supplied at run time with --parameters= (see below).

Compiling for RISC-Vlink

The base command to produce RISC-V vector code is:

iree-compile mobilenet.mlir -o mobilenet_rv64.vmfb \<br>--iree-hal-target-device=local \<br>--iree-hal-local-target-device-backends=llvm-cpu \<br>--iree-llvmcpu-target-triple=riscv64 \<br>--iree-llvmcpu-target-abi=lp64d \<br>--iree-llvmcpu-target-cpu-features=+m,+a,+f,+d,+c,+zvl512b,+v

The flag that matters most on RISC-V is --iree-llvmcpu-target-cpu-features ,<br>which specifies the ISA. +m,+a,+f,+d,+c is rv64gc, +v enables RVV 1.0, and<br>+zvl512b declares the minimum vector register width (VLEN) — 512 bits here.<br>The zvl<br>value should match the target hardware's actual VLEN — 512 for the QEMU<br>configuration used below, 256 on a device such as a SpaceMiT X60 — since a<br>mismatch leaves the vector units underutilized. VLEN is the key RISC-V knob: it<br>drives LLVM's vector codegen and the tile sizes IREE selects for data-tiling<br>(more on that below).

The remaining flags are the optimization knobs. None of them are RISC-V-specific,<br>but they are where the performance comes from, so they are layered on top of the<br>base command.

Data-tilinglink

--iree-opt-data-tiling repacks matmul-shaped operations into a tiled mmt4d<br>layout that maps cleanly onto the vector unit. It is off by default; most models,<br>especially matmul-heavy models, benefit from this. On RISC-V the tile shape<br>depends on<br>VLEN, so the +zvl*b value chosen above also determines the produced layout. The<br>data-tiling walkthrough<br>and mmt4d blogpost<br>cover the mechanism in detail.

im2col for convolutionslink

--iree-global-opt-use-im2col-for-convs=true rewrites convolutions as im2col plus<br>matmul, so that...