GitHub - AspadaX/tiny-llama: A minimal LLaMA model inference engine built purely in Rust · GitHub
/" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
AspadaX
tiny-llama
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>8 Commits<br>8 Commits
src
src
.gitignore
.gitignore
Cargo.lock
Cargo.lock
Cargo.toml
Cargo.toml
LICENSE
LICENSE
README.md
README.md
demo.gif
demo.gif
View all files
Repository files navigation
tiny-llama
A minimal LLaMA-style inference engine for MiniCPM5-1B, built purely in Rust.
I built this project to study the fundamentals of large language models. Every understanding is rephrased directly in the code — dense, but organized. My hope is that it helps you learn too.
Inspired by tiny-vllm.
Features
CPU-only inference — no GPU required
No KV-cache — focuses on the core model mechanics
Single-file implementation — src/main.rs (~1800 lines)
TUI visualization — real-time view of how the model "thinks"
What's implemented
Component<br>Details
Embedding<br>Token ID lookup via embedding table
RoPE<br>Rotary position embeddings
Attention<br>Multi-head attention with GQA (Grouped Query Attention)
MLP<br>SwiGLU activation function
Normalization<br>RMSNorm
Residual connections<br>Standard skip connections after attention and MLP
Getting Started
1. Download the model
# Download MiniCPM5-1B from HuggingFace<br># Put the model files in a directory, e.g. ./models/minicpm5-1b/<br># The directory should contain:<br># - config.json<br># - model-00000-of-00001.safetensors<br># - tokenizer.json
2. Build and run
" """>cargo run --release -- "" ""
Example:
cargo run --release -- ./models/minicpm5-1b "What is artificial intelligence?"
Press q to quit early.
Code Walkthrough
The best way to read this code is top-to-bottom in src/main.rs:
Tensor ops — A minimal TinyTensor wrapper over candle tensors with matrix multiply, reshape, transpose, softmax, etc. I will try to write the Maths operations by hand.
Model loading — ModelConfigurations, LlamaModel, and TransformerBlock structs that map directly to the safetensors keys.
predict_next_token — The inference loop through every transformer layer. Each operation is annotated with shape comments and timing.
TUI — Ratatui widgets render attention maps, tensor shapes, and logits in real time.
License
MIT — see LICENSE
About
A minimal LLaMA model inference engine built purely in Rust
Topics
rust
machine-learning
ai
deep-learning
neural-network
algorithms
transformers
tui
llama
attention-mechanism
inference-engine
minicpm
Resources
Readme
License
MIT license
Uh oh!
There was an error while loading. Please reload this page.
Activity
Stars
stars
Watchers
watching
Forks
forks
Report repository
Releases
No releases published
Packages
Uh oh!
There was an error while loading. Please reload this page.
Contributors
Uh oh!
There was an error while loading. Please reload this page.
Languages
Rust<br>100.0%
You can’t perform that action at this time.