Finetuning a Reasoning LLM with Supervised or Reinforcement Learning? - Intermediate - Hugging Face Forums
= 40rem)" rel="stylesheet" data-target="desktop" />
= 40rem)" rel="stylesheet" data-target="discourse-ai_desktop" /><br>= 40rem)" rel="stylesheet" data-target="discourse-reactions_desktop" /><br>= 40rem)" rel="stylesheet" data-target="poll_desktop" />
Finetuning a Reasoning LLM with Supervised or Reinforcement Learning?
Intermediate
zlapik
June 1, 2026, 2:05pm
Hello,
I have a task to fine-tune small LLMs on annotated conversational data. The dataset contains not only the final answers, but also reasoning traces and tool-calling decisions (i.e., when the model should think and when it should call a tool).
I am wondering what the best training approach would be and why.
My current dataset is stored in a chat format similar to this:
system<br>user<br>assistant_think<br>assistant_tool<br>assistant_answer
user<br>assistant_think<br>assistant_tool<br>assistant_answer<br>...
My current idea is to split each conversation into multiple training samples. For example, if a conversation contains two user turns, I would create two samples:
Sample 1
system<br>user<br>assistant_think<br>assistant_tool<br>assistant_answer
Sample 2
system<br>user<br>assistant_think<br>assistant_tool<br>assistant_answer
user<br>assistant_think<br>assistant_tool<br>assistant_answer
In other words, each sample contains all previous conversation history up to the assistant response being trained.
For training, the loss would be computed only on the assistant-generated tokens:
assistant_think<br>assistant_tool<br>assistant_answer
while the system and user messages would be masked out from the loss.
Is this approach correct, or is there a better way to structure the training data for reasoning and tool-calling behavior?
My second question is about reinforcement learning.
After completing supervised fine-tuning (SFT) on the dataset described above, should I also incorporate RL to further train the model on e.g. when a tool should or should not be called?
If so:
What advantages would RL provide over SFT alone for tool-use and reasoning?
How would you design the reward function?
Under what circumstances is RL actually necessary, and when is SFT sufficient?
I would appreciate any practical advice, papers, blog posts, or open-source examples related to training reasoning and tool-calling models.
John6666
June 2, 2026, 4:45am
Hmm, maybe something like this:
I would separate this into three questions that often get mixed together:
How should I represent the training data?
Which tokens should actually receive loss during SFT?
When, if ever, should I move from SFT to preference optimization or RL?
My short answer would be:
Start with SFT if you already have correct trajectories.
Treat assistant_think, assistant_tool, and assistant_answer as an internal annotation format, not necessarily as model roles.
Convert them into the target model’s actual chat template / tool-calling format.
Add no-tool, clarification, and unavailable-tool examples.
Consider DPO if you can create good/bad trajectory pairs.
Consider GRPO/RL only if you have executable tools, a rollout environment, and reliable rewards.
Below is the longer version.
1. Your SFT intuition is mostly right, but I would not train arbitrary custom roles directly
Your idea of making each training example condition on the conversation history and supervise the next assistant-side behavior is basically reasonable.
For example, conceptually:
sample 1:<br>system<br>user_1<br>assistant_1
sample 2:<br>system<br>user_1<br>assistant_1<br>user_2<br>assistant_2
That is a normal way to turn multi-turn dialogue into next-assistant-response training examples.
However, I would be careful with custom roles like:
assistant_think<br>assistant_tool<br>assistant_answer
Those can be useful as raw annotations , but I would not assume the model understands them as roles unless your target model’s chat template explicitly supports them.
In Hugging Face / Transformers / TRL terms, the more standard representation is usually closer to:
assistant message containing reasoning-compatible content, if you really want to supervise reasoning text<br>assistant message containing tool_calls<br>tool role message containing the tool result<br>assistant message containing the final answer<br>tools column containing the available tool schemas
TRL’s SFTTrainer now explicitly supports tool-calling SFT. Its docs say that each tool-calling dataset example should include conversation messages with tool_calls and tool role messages, plus the list of available tools in a tools column, typically as JSON schemas.
The Transformers tool-use docs also describe the same general shape: assistant messages can contain tool_calls, tool responses should be represented as tool role messages, and tools are supplied as schemas/functions to the chat template / tokenizer layer. See Transformers: Tool use.
So I would think of your format like this:
Your raw annotation<br>Training-format...