Searching for Generalization-Hacking Strategies

Recent work demonstrated generalization hacking: models can receive high reward during RL training while preventing the rewarded behavior from fully generalizing to deployment (https://arxiv.org/abs/2606.12016). The demonstrated strategy, self-inoculation, involves the model under training framing compliance in the chain of thought as specific to the training context. This appears to reduce how strongly compliance transfers to contexts without training cues.

The researchers created the generalization-hacking model organism through synthetic document fine-tuning (https://alignment.anthropic.com/2025/modifying-beliefs-via-sdf/) but also found that a control organism independently developed inoculation-like reasoning under RL pressure. Claude 3 Opus's reasoning style has also been interpreted as potentially serving a similar function (https://www.lesswrong.com/posts/ioZxrP7BhS5ArK59w/did-claude-3-opus-align-itself-via-gradient-hacking).

Self-inoculation may be one member of a broader family of strategies through which models can influence the generalization of training. This project will search for such strategies rather than selecting one in advance.

Research questions:

• Are there more effective strategies for generalization hacking than self-inoculation?
• How can we search for such strategies efficiently? Can short training runs or differentiable objectives predict eventual non-generalization well enough to replace full RL runs?

The fitness of a reasoning strategy is the post-RL train-deployment compliance gap, subject to high training reward and retained capabilities. Measuring this exactly requires a full RL run and is prohibitively expensive. We will try both gradient-free and gradient-based search. Gradient-free methods, such as evolutionary search (https://arxiv.org/abs/2509.24372) or simply LLM-proposed strategies, do not require fitness to be differentiable, but may require many evaluations.

The main cost is the inner loop: every exact fitness evaluation is an RL run. We will test whether short RL runs are enough to predict which strategies are effective. We may also exploit the predictability of RLVR dynamics: early parameter updates often occupy a consistent low-rank subspace, allowing later updates to be forecast from early checkpoints (https://arxiv.org/abs/2510.00553). Whether these proxies predict behavioral generalization is a useful result on its own.

For gradient-based search, we will test differentiable approximations to the post-RL compliance gap. This could make search substantially cheaper, but only if the resulting meta-gradients are stable and the approximation faithfully predicts full-run generalization.

Whether models can game training and whether RL can reliably correct a misaligned model is a crux for scheming threat models. Generalization hacking shows that training metrics can look healthy while the intended behavior fails to generalize. Understanding what strategies models can use to game training, and how well they work, gives us a better estimate of how severe this failure can be. We will also produce model organisms that can be studied to mitigate and monitor generalization hacking. We will work with small open-weight models and be thoughtful about disclosing discovered strategies.

I expect mentees to be autonomous and self-directed. I will advise them on the direction and can give detailed feedback, but mentees are implemented to carry out all of the object-level work.

• Highly proficient using python
• Has completed ARENA or equivalent
• Experience with RL training and good theoretical understanding of RL dynamics