pantheon/workflows/ai/w02-tiny-llm at main · temple-compute/pantheon · GitHub
//files/disambiguate" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
//files/disambiguate;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Uh oh!
There was an error while loading. Please reload this page.
temple-compute
pantheon
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
FilesExpand file tree
main
/w02-tiny-llm<br>Copy path
Directory actions
More options<br>More options
Directory actions
More options<br>More options
Latest commit
History<br>History<br>History
main
/w02-tiny-llm<br>Copy path
Top
Folders and files<br>NameNameLast commit message<br>Last commit date<br>parent directory<br>..<br>scripts
scripts
README.md
README.md
conda_env.yaml
conda_env.yaml
pyproject.toml
pyproject.toml
workflow.yaml
workflow.yaml
View all files
README.md<br>Outline<br>W-29 · Tiny LLM From Scratch - TinyStories Pretrain & Sample
Overview
A fast, small-data companion to W-28: pretrains the same<br>from-scratch decoder-only Transformer, but on<br>TinyStories (short, simple<br>GPT-generated children's stories, a couple hundred MB) instead of the Pile (~900GB). It exists<br>for people who want to see a from-scratch LLM learn something legible in minutes on a laptop,<br>without the Pile's disk/bandwidth requirements. It skips instruction tuning and eval entirely.<br>The deliverable is a pretrained checkpoint plus a sample generation proving it learned to<br>continue a story prompt.
Pipeline
clone_repo (local)<br>├─► prepare_tinystories_train / prepare_tinystories_val ──► tinystories_{train,val}.h5<br>│ └─► train_pretrain_tiny (local) ──► tinystories_pretrained.pt<br>│ └─► sample_story (local) ──► tinystories_sample.txt
Quick start
# Install the horus-runtime and plugins (one time)<br>uv sync
# You can install UV with this command if you don't have it yet:<br>curl -LsSf https://astral.sh/uv/install.sh | sh
# Otherwise, you can install the horus-runtime and plugins with pip:<br># pip install horus-runtime horus-environments
# Run the workflow<br>uv run horus run workflow.yaml
Outputs land under workflow_results/: repo/ (the cloned training repo), data/ (tokenized<br>TinyStories), ckpts/tinystories_pretrained.pt (checkpoint), and<br>samples/tinystories_sample.txt (a greedy continuation of "Once upon a time,").
Tools & Dependencies
train-llm-from-scratch (cloned at<br>runtime by clone_repo). Pure PyTorch, no transformers/trl/peft. Only<br>scripts/pretrain_base.py and scripts/chat.py are used from it.
TinyStories. Downloaded via the<br>datasets library.
scripts/prepare_tinystories.py (this workflow's own script, not from the cloned repo),<br>tokenizes TinyStories with tiktoken into the same flat-token HDF5 layout<br>(data_loader/data_loader.py's {"tokens": int32[N]}) that pretrain_base.py expects.<br>The cloned repo's own scripts/prepare_pretrain_data.py is hardcoded to the Pile, so this<br>workflow can't reuse it directly.
torch, numpy, h5py, datasets, tiktoken, provisioned via conda_env.yaml<br>(micromamba) by the conda_python_environment (horus-environments) plugin executor.
Horus Configuration
Every task runs target: {kind: local}, either on the bare shell executor (just clone_repo)<br>or the conda_python_environment executor. prepare_tinystories_{train,val} take repo_dir as an input purely to give<br>every task in this workflow one shared root (clone_repo), even though the tokenizer script<br>itself doesn't read from the checkout.
Input / Output
Input : none. Self-contained; clones its own source repo and downloads TinyStories from<br>Hugging Face.
Output :
ckpts/tinystories_pretrained.pt. Self-describing checkpoint (embeds the resolved model<br>config, training step, and metrics).
logs/pretrain.jsonl. One JSON record per logged training step.
samples/tinystories_sample.txt. A greedy, raw (non-chat-template) continuation of<br>"Once upon a time," from the trained model.
Parameterization
Dataset size : prepare_tinystories_train/prepare_tinystories_val's --max_docs (default<br>20,000 / 2,000) caps how many stories get tokenized, raise it (or drop the flag entirely) for<br>more data at the cost of longer prep time. Full TinyStories is ~2.1M train stories.
Model/step-count scale : train_pretrain_tiny loads...