Karpathy's LLM Pedagogy

Hub · Overview<br>Karpathy's LLM Pedagogy

This wiki covers Andrej Karpathy's published teaching corpus on language models — seven open-source repositories and a nine-lecture YouTube series ("Neural Networks: Zero to Hero"). Together they trace the technical lineage from "what is backpropagation" through to "here is a working reproduction of GPT-2 (124M)."

The corpus is unusually coherent. The same patterns and abstractions recur across repos — Block, MultiHeadAttention, configure_optimizers, estimate_mfu, from_pretrained — at progressively bigger scales. Reading any one repo in isolation works, but reading them in order shows you the underlying ideas being refined.

Reading guide

If you're starting from zero and want the full arc, the order is:

zero-to-hero-arc<br>The lecture map. Read this first.

repos/micrograd<br>Scalar autograd. The conceptual root.

backpropagation and value-class<br>The algorithm and its data structure.

repos/makemore<br>First real LMs. Bigram → MLP → ... → Transformer.

repos/ng-video-lecture<br>Character-level GPT on Tiny Shakespeare.

repos/nanoGPT<br>Production-grade GPT-2 implementation.

repos/build-nanogpt<br>Faithful GPT-2 reproduction with every optimization.

repos/llama2-c<br>Llama 2 in PyTorch + pure C inference. The "modern" architecture.

repos/llm-c<br>Same training task as build-nanogpt, in pure C/CUDA.

If you want to learn a specific concept, jump to the concept page; each one cross-references the repos that demonstrate it.

The architecture, in pieces

The transformer architecture as Karpathy teaches it, broken into independent pieces:

TopicPage

The repeating unittransformer-block<br>Information mixing across positionsattention<br>Stability mechanism for deep stacksresidual-connections<br>Per-layer normalizationlayernorm-vs-rmsnorm<br>Per-position nonlinearitygelu-and-swiglu<br>Positional information (GPT-2 vs Llama)rope<br>Vocabulary and embeddingtokenization, character-vs-bpe<br>Embedding-unembedding sharingweight-tying

Training, in pieces

TopicPage

Gradient computationbackpropagation, value-class<br>Parameter updateadamw<br>Initializationweight-init<br>Learning rate over timelearning-rate-schedules<br>Batches and effective batch sizegradient-accumulation, dataloader<br>Numerical precisionmixed-precision-and-mfu<br>Keeping training alivetraining-stability<br>Downstream evaluationhellaswag-eval

Inference

TopicPage

Token selectionsampling<br>Generation accelerationkv-cache<br>Pure-C runtimerepos/llama2-c

Three "model families" to compare

The corpus contains three subtly different transformer architectures, useful to compare against each other:

Component<br>GPT-2 ng-video-lecture, nanoGPT, build-nanogpt, llm.c<br>Llama 2 llama2.c<br>makemore Transformer

Normalization<br>LayerNorm<br>RMSNorm<br>LayerNorm

Positional<br>Learned embedding<br>RoPE<br>Learned embedding

Activation<br>GELU<br>SwiGLU<br>GELU

Tokenizer<br>BPE (50257)<br>SentencePiece BPE (32000)<br>character-level

Attention<br>Multi-head<br>Grouped-query<br>Multi-head

Same skeleton, different organs. Once you know the skeleton (the transformer block wrapped in residuals and a stack), swapping organs is straightforward.

What's not in this wiki

Things outside the scope of the corpus:

Post-training (SFT, RLHF, DPO)<br>None of these repos do instruction tuning or alignment. nanochat does, but it's not in the corpus.

Model parallelism beyond DDP<br>No tensor parallelism, no pipeline parallelism. llm.c has ZeRO-1 optimizer sharding but no model sharding.

Multimodal<br>Text-only throughout.

MoE<br>Dense models only.

In scope: dense, decoder-only, pretraining + base inference, up to GPT-2 / Llama 2 scale. Within that scope it's the most complete teaching resource available.

Cross-reference conventions

Every page in this wiki uses markdown reference links: [name](name.md) for concepts, [name](repos/name.md) for repos. The link text is usually the unqualified name; the path tells you whether it's a concept or a repo page.

For agents post-processing this wiki: every page is a self-contained topic that can be rendered as a single HTML page. Internal links between pages are the primary structural signal of the wiki graph. The concepts/ flat layout was rejected in favor of having concepts at the wiki root and repos in a subdirectory — concepts are first-class citizens, repos are case studies that ground them.