Language Model Builder — Build models. Understand AI.Train your own language model

One of the best ways to understand technology, especially when it seems like magic, is to build a small version yourself. Language Model Builder explains the foundations behind mainstream AI tools and lets you build models from scratch. Choose a tokenizer and training data, watch the loss fall during pre-training, teach the model to answer questions, and end up chatting with something that you made.

Some assembly required, manual included.<br>Language Model Builder has two parts: It offers both a textbook-style introduction to the foundations of how language models are built as well as beautifully designed tools for actually building a model.

Interactive textbook<br>A short textbook with interactive playgrounds<br>I often find myself explaining how language models are built and trained to friends and family. I finally wrote the introduction I always wanted: It&rsquo;s friendly to non-programmers, contains many interactive examples, and covers all the fundamentals without treating the reader like a child or a machine learning expert.<br>Interactive from the first page<br>Run the experiments yourself: build a tokenizer, move through embedding space, inspect attention, and watch a tiny model learn.

Go as deep as you like<br>No coding, machine learning, or math experience required. Where useful, I’ve added opportunities for those curious to dive a little deeper.

No artificial additives<br>The educational content is only as long as it needs to be, without generated filler content.

Native training workbench<br>MLX

Train a GPT on your Mac.<br>Run the entire model creation pipeline directly on your Mac, with hardware-accelerated MLX, without installing any additional tools. Every major step comes with beautifully crafted tools that you&rsquo;ll love.<br>The whole training pipeline<br>Pre-train with a data set of your choice, sample the base model, run supervised fine-tuning and direct preference optimization, then chat with the model you made.

See the run as it happens<br>Follow loss, throughput, validation, saved samples, and checkpoint history without stitching together scripts. It’s really fun to watch your model become more and more capable.

Fully local & free<br>Resume from a saved checkpoint tomorrow. Your models, datasets, and training history stay on your Mac. No accounts, no cloud, no bill.

Part I &middot; A friendly introduction<br>Understand language models in about 90 minutes<br>I tried very hard to make the introduction as tight as possible: It covers all the major components, yet doesn&rsquo;t ask you to read hundreds of pages. The app is a living project, I&rsquo;ll add more content based on feedback and the (usually quick!) evolution of the landscape.

Language Model Builder&hearts;<br>LanguageModel Builderthe·little·cat·played·in·the·warm·sun

Contents<br>Language modelsnext-word prediction, n-grams, RNNs<br>Tokenizationhow text becomes numbers<br>Embeddingshow numbers acquire meaning<br>Attentionhow words take in their context<br>The transformerthe architecture inside every modern model<br>Training dataweb crawls, filtering, deduplication<br>Loss & gradient descenthow a model actually learns<br>Fine-tuningteaching by example (SFT), then by preference (DPO)

ToolsAccelerated training with Apple&rsquo;s MLX

II<br>Build a model with beautiful tools<br>Language Model Builder offers well-crafted tools you&rsquo;ll love. Dataset selection, pre-training, sampling, supervised fine-tuning and direct preference optimization; they all come with approachable and powerful tools. Everything runs locally, in-process, on your Mac.

01 &middot; Training<br>A training run you can watch.<br>Pre-training, supervised fine-tuning, direct preference optimization, and sampling — all in-process, on your Mac. Watch the loss curve fall, teach the model your taste and preferences, then compare different model settings.<br>Pre-trainingSFTDPOSampling

02 &middot; Data<br>A catalog of good data.<br>Curated datasets for both pre-training and fine-tuning, each explained and one click away: bigger corpora for pre-training; friendly chats, human-written instructions, and math problems for fine-tuning. Also, obviously, full support for your own data sets.<br>Pre-training dataFine-tuning data

03 &middot; Chat<br>Talk to your creation<br>If you&rsquo;ve never had the pleasure, it&rsquo;s hard to describe just how fun it is to talk to a model that you&rsquo;ve made. Language Model Builder allows you to chat in a pretty environment that also allows deep inspection - like token probabilities and tokens not chosen, for instance.<br>ConversationX-ray

&times;Pre-trainingSFTDPOSamplingPre-training dataFine-tuning dataConversationX-ray

What kind of model can I actually build with this?Using the default settings, you’ll get a model that writes coherent, grammatical multi-paragraph text in as little as a day. On a MacBook Pro M5 Max you could train a GPT-2-small-class model (~100–150M parameters on a few billion tokens) in about a week. It might be...