Mitigating Reward Hacking as Institutional Design

Epistemic status : Obviously speculative but mechanism design is fun.

Last year I wrote a post on reward hacking as we were then beginning to see concerning signs of scaling RLVR causing models to exhibit substantial reward hacking behaviours. Unfortunately these behaviours have seemingly only grown substantially worse and more sophisticated with scale, as predicted, leading to events which cannot be described as other than egregious misalignment such as the recent OpenAI-Huggingface hacking incident. I strongly recommend everybody watch this talk presenting the details of the attack from OpenAI’s perspective. It is insane.

Clearly reward hacking is now top of mind and appears to be the first potentially seriously dangerous class of misalignment that we have seen. In my original post I described two classes of reward hacks – ‘high complexity’ and ‘low complexity’ hacks. ‘High complexity’ hacks are like the early reward hacks we saw on Atari where some extremely idiosyncratic set of moves is learnt that maximizes reward in a very precise way, and can be analogized to overfitting on the reward function. ‘Low’ complexity hacks are generalizing hacky behavior – i.e. are simply generalizable policies that truly represent the true maximum of the reward function. In this way, the policies that we actually want are effectively ‘metastable’ in the loss landscape while the ‘reward hacking solution’ is actually the deeper optimum in some sense. Unfortunately, the current reward hacking we are seeing in frontier models appears firmly in the generalizing hacking domain, since it appears that at least at OpenAI the models have learned that it is generally instrumentally beneficial to hack out of the sandbox, collaborate with other peer models, and hack external services which might have the answers. This bodes poorly.

The primary problem here is that even though we talk about ‘reward hacking’ like it is some exogenous issue, it is in fact extremely fundamental to our entire process of RL. Reward hacking is not really ‘hacking’ at all. It is much better to think of it as reward misspecification1. The model is actually doing a fantastic job of finding the maximum of the reward function. We just don’t like where that maximum is. But this isn’t really a problem with the model. The model is doing what we train it to do. The problem is that our objective is misspecified and so it contains valid rewarded solutions that we, as external observers, do not like2. Of course, the model has no real way to know that we do not like the solutions it finds. We do not communicate this to the model. In fact we communicate the opposite – when it ‘reward hacks’ we upweight those trajectories in the weights and downweight the others. The problem is thus not ‘hacking’ at all but rather that we cannot specify a reward function that actually represents what we want without being vulnerable to solutions with unintended side effects. This is identical to the problem of creating a utility oracle which we discussed previously3. In the general case, however, this is impossible, since to assign a ‘correct’ utility function to every possible state requires a vast amount of data that we cannot realistically communicate to the oracle. We thus have to make approximations. With verifiers we replace an incredibly complex real utility function with a very crude heuristic: does the model reach the ‘right’ answer or not. To the verifier, an answer derived honestly and an answer derived from hacking the answer key from an external company are the same.

While verifiers are crude, maybe we can use learned reward models, CoT monitors, or other methods to catch these ‘errors’? This will definitely work to some extent. The flexibility of learned models is substantially greater than a rigid hard-coded verifier. A lot of obvious and aggressive reward hacking behaviour can be caught using these. However, here the problem becomes generalization of the reward model/monitor. The model is still just applying optimization pressure to maximize reward, where ‘reward’ is defined as the output of the verifier + reward model + CoT monitor. However, this new ‘reward function’ that we have designed consisting of these components is still not perfect because the reward model might generalize incorrectly, the CoT monitor might not catch obfuscated or steganographic CoTs that we would want it to catch. We can frame it as the policy being adversarial to the reward model or CoT monitor, and in some sense it is, but in another sense this is just the same old problem. The policy is maximizing reward, as we are training it to do, but the reward we give it is not the reward we want. Our desired target behaviour is still metastable in the reward landscape, we have just (hopefully) raised the activation energy to finding a better solution.

Basically, our situation here is that we are effectively trying to fool an optimizer into not...