how we monitor our rl training runs - castform blog

start training book demo

pricing docs learn blog careers start training book demo

at castform, we’ve babysat well over a thousand rl post-training runs. in the midst of all that, the way we’ve been tracking our runs has changed quite a bit, as we accumulated scar tissue. this post is an attempt to write/codify some of the intuitions we’ve developed over time. for both internal and external use 🙂

if there’s one takeaway we hope you’d take away from this: don’t just look at the mean reward curve.

reward curves lie all the time

the first thing anyone looks at during an rl run is whether the mean reward curve goes up. unfortunately, that’s also the metric that’s the most misleading. particularly because it’s just one number averaged over a batch of prompts. just because this number is climbing doesn’t mean things are going well. some examples:

the model’s simply hacking the reward e.g. the output is degenerating in ways the reward model seems to like (extra emojis, padding, a particular format type, etc.)

one subset of your prompts is collapsing while another subset is improving enough to mask that

the environment itself is feeding back flawed observations that the model is exploiting

all of the above is pretty hard to debug from the mean reward curve. so while the mean reward curve is a good sanity check to ensure learning is happening, monitoring jobs shouldn’t end there.

mean reward climbing steadily — but this alone masks a lot of what’s actually happening.

read the model’s completions during training

we get the biggest bang for the buck by reading sampled completions from the model during training. not summary stats but the raw text itself. often, we see ourselves sorting completions/prompts by reward score to see if the reward is deserved or whether it’s just a hack. same for the inverse, we look at the lowest scoring reward samples to see if it’s an actual failure mode or if the reward function is just misconfigured somewhere. every reward hack we have caught has been this way.

viewing tool calls and final answers across rollouts, sorted by reward score.

inspect what’s going on in the environment

we liberally log what’s going on in the environment the model is being trained within. this starts with things like tool calls → why tool calls might be failing, is the output mangled, etc.

it’s also really useful to do this in cases where reward functions are llm judges → we make sure to log reasoning traces/outputs and inspect them as exhaustively as possible to catch abnormalities and patch rubrics.

inspecting llm judge outputs against the reward rubric to catch scoring abnormalities.

look at the prompt types that are underperforming

averages hide a lot of the variance in performance, so it’s important to disaggregate prompt types to see what types of prompts are doing well and what aren’t. so, we find ourselves often sorting prompts by reward → and seeing which ones are doing poorly and which ones are regressing over training.

diversity >> mean

the second form of intuition we’ve developed is around the importance of diversity. it’s easy to benchmax a model to do very very well on some subset of the dataset and lose its ability to really solve other sorts of problems. this has implications for when the model is actually deployed. the real world is messy and you want the model to be able to handle out-of-distribution tasks gracefully.

track max@k

instead of mean reward, the quantitative reward we actually track is max@k. max@k is the highest reward achieved across k sampled completions for the same prompt - i.e. how good the model’s best attempt is when you give it k tries.

the gap between mean@k and max@k is something we track quite religiously, because it’s a proxy for how much more exploration headroom the model still has. when max@k starts collapsing towards mean@k, it often means the model isn’t exploring and might have overly committed to a narrow behavior type. even if the mean reward looks fine, this means you might have lost the diversity that future learning depends on.

tracking max@k alongside mean@k — the gap between them is a signal of how much exploration headroom remains.

track the types of solutions the model is producing

beyond just tracking the numbers, it’s also important to track the shape of the solutions the model is generating. we ideally want the model to be attempting & discovering new strategies to get to the correct answer. for agentic/reasoning tasks in particular, when you notice solution diversity narrowing, it’s an early warning. a model that can only solve problems in a particular way has likely overfit to the reward function and will likely generalize poorly. keeping an eye on the solution type helps catch this issue early.

build hard, ood evals

finally, the last suggestion is around making sure we don’t fool ourselves. if progress is only measured based on training reward distribution,...