As an interesting experiment I wanted to learn how to teach a tiny local llm a new domain by doing Continued Pretraining (CPT) on domain specific data. This article is a write-up of my experiences from using Unsloth to train qwen 3 4B to act as a travel advisor for a fictional city.<br>">

As an interesting experiment I wanted to learn how to teach a tiny local llm a new domain by doing Continued Pretraining (CPT) on domain specific data. This article is a write-up of my experiences from using Unsloth to train qwen 3 4B to act as a travel advisor for a fictional city.<br>" />

Teaching a Local LLM a New Domain

Teach Me Cool Stuff

Teaching a Local LLM a New Domain

Published: 20 Aug, 2026

Author Torgeir Helgevold

As an interesting experiment I wanted to learn how to teach a tiny local llm a new domain by doing Continued Pretraining (CPT) on domain specific data. This article is a write-up of my experiences from using Unsloth to train qwen 3 4B to act as a travel advisor for a fictional city.

Continued Pretraining (CPT)

As the name suggests, CPT is a continuation of the model’s initial training, typically on a specific domain to allow the model to specialize on top of what it already knows.

Normally, full CPT on a model of any size would be impractical on consumer hardware due to high VRAM requirements. However, there is a clever workaround called LoRA (Low-Rank Adaption). The basic idea behind LoRA is that the original model weights are kept frozen while small trainable LoRA adapters are attached to some of the layers of the model. During training, we only update the parameters in the LoRA adapter. In practice this means we only have to touch a fraction of the full set of parameters.

Example: I am working with qwen 3 4B, which has 4 billion trainable parameters. However with my current LoRA configuration, I am only targeting 66 million parameters - 1.6% of the total number of parameters!

To do the actual LoRA based CPT training I am using a framework called Unsloth. I have included the full source here in case you are interested in checking it out.

Domain

The domain can be anything, but I decided to try to teach the model to act as a travel advisor for a fictional city called Awesomeville in the country of Greatness. Everything about the city is made up of course, but the town has its own subway with multiple historical sites located near the subway stations.

The goal of this exercise is to teach the model to reason about the subway map and belonging historical sites.

As an example I want the model to reliably answer questions like: What is the subway route from Museum of Greatness History to Founder's Square?

As an illustration, I have added a snapshot of the subway map below:

Awesomeville’s three subway lines connect at Central Station.<br>Select the map to view it full-size.

As you can see there are three subway lines in the city (Green, Blue and Gold) with various stations located near historical sites in town.

The map also shows that the lines connect through central station, so a travel advisor needs to be able to reason and recommend multi-line routes.<br>Some of the key scenarios are listed below:

Same line travel (e.g. Blue line stations only)

Single transfer through Central station (e.g. Starting on Blue line and transferring to Green line)

Connect trips with multiple transfers (e.g. Blue -> Green -> Gold)

Corpus

Create Data

As expected, the most time-consuming part of this project by far was defining the corpus, the collection of data used to train the model. Since the entire universe of Awesomeville is fictional, all data had to be synthesized. Using an agent for this is of course the most practical solution these days, but it’s not as easy as just asking an agent for a perfect training corpus.

I discovered a few pitfalls when synthesizing data using agents.

One thing to keep in mind is that agents often create text generators, which may lead to templated language with lots of repetition from shared intros and fragments. Too much common phrasing around a few variables like station names may make it harder for the model to learn the subtle differences.

Another issue is that agents generate data so fast that it’s easy to lose track, and before you know it, your corpus has grown to 10k entries of questionable quality.

One of the key goals of this exercise was to come up with a set of training data that would enable the model to reason. I wanted to avoid a situation where the data consists of a large amount of specific route examples since this tends to lead to route memorization and likely poor generalization.

Structuring the Data

Experimental Phase

The first draft of my training data was sort of a disaster. Initially, I felt like I was moving in the right direction but before I knew it, I had generated a 10k bloated dataset that relied far too much on memorization of specific route scenarios. Performance would often be poor when unseen examples were introduced. The large dataset...