Notes on Midtraining - by Cameron R. Wolfe, Ph.D.

Deep (Learning) Focus

SubscribeSign in

Notes on Midtraining<br>Creating better specialized LLMs with midtraining and continual pretraining...

Cameron R. Wolfe, Ph.D.<br>Aug 10, 2026<br>∙ Paid

32

Share

(from [2, 3, 4, 6, 11])<br>The training process for large language models (LLMs) has historically been composed of two high-level stages: pretraining and post-training. Pretraining exposes the LLM to a large volume of diverse textual data, while post-training focuses on refining and aligning model behavior via techniques like supervised fine-tuning (SFT), reinforcement learning (RL), and more. Recently, LLM training pipelines have become increasingly complex due to the addition of intermediate training stages between pretraining and post-training. In particular, three new stages are commonly encountered in the literature:<br>Continued Pretraining (CPT): an extended period of pretraining that begins from an existing pretrained checkpoint, usually with the goal of specializing the model toward a particular domain or data distribution.

Midtraining: an intermediate stage between general pretraining and post-training that continues the pretraining process on a more curated data distribution, often by annealing the data mixture toward higher-quality, domain-specific, reasoning, or instruction-like data.

Long-Context Training: a dedicated training period that extends the model to longer sequence lengths after initially training on shorter sequences to improve training efficiency.

In this overview, we will focus on the first two concepts outlined above, covering a wide variety of industry tech reports and research papers that propose practical strategies and best practices for CPT and midtraining. Notably, these techniques are highly similar in implementation—both typically continue LLM pretraining on large-scale textual data using next-token prediction or related objectives. As a result, the boundary between CPT and midtraining is somewhat blurry, and the terminology is not always used consistently across papers. However, these are usually framed as two different concepts in practice:<br>CPT commonly refers to taking an existing pretrained model—oftentimes from another research group or organization—and continuing to train it on data that specializes the model toward a domain or application of interest.

Midtraining more often refers to a planned stage within the original LLM training pipeline that bridges general pretraining and post-training. Rather than pretraining on a fixed distribution until completion, the data mixture is progressively adapted over time to emphasize different data distributions.

As we will see, CPT and midtraining are powerful tools for building domain-specialized LLMs. However, using these techniques effectively requires rigorous empirical analysis and evaluation to inform proper choices around data quality, mixture design, timing, and duration. To understand how to properly tune these training stages, this post will analyze recent training efforts in detail and extract relevant best practices for successfully configuring midtraining and CPT.<br>Preliminaries. There are several background concepts that will help to better understand the papers included in this post. Links to further reading on each of these topics have been provided below for reference:<br>LLM pretraining and scaling laws.

Data curation for pretraining.

Supervised finetuning (SFT).

Reinforcement learning (RL).

Join many others who use Deep (Learning) Focus to understand AI research. Consider a paid subscription if you would like to help support the newsletter.

Subscribe

Characterizing Datasets and Building Better Models with Continued Pre-Training [1]

Although LLMs are proficient at a wide variety of tasks, these models can fall short in solving tasks that are not meaningfully represented in their training distribution. If we know during training that our model will be deployed in a particular domain, we can often improve performance by performing CPT on domain-relevant data. In [1], the authors present a practical recipe for efficiently tuning CPT by using shorter and cheaper proxy experiments to identify the best datasets, learning rates, training durations, and data mixtures.<br>“Running LR sweeps at the 1B-10B token regime is a fast way to identify which datasets enhance model performance. This allows us to remove ineffectual datasets and eventually mix the beneficial datasets and train them for longer periods, making CPT an efficient tool for identifying useful datasets.” - from [1]

The central observation in [1] is that short CPT runs can accurately predict the effectiveness of much longer runs. Rather than immediately running large-scale CPT experiments with various datasets, we can first characterize each dataset using a smaller proxy run. More specifically, we can determine which datasets are worth including in CPT by:<br>Performing a short CPT run.

Measuring whether training...