Context: What Is Orthogonalization?

Orthogonalization aka ablation aka abliteration aka the technique from Refusal Is Mediated by a Single Direction is the most common technique used in practice to remove refusal from open weight models. For removing refusal, it is more sample efficient and more tolerant to low quality data than other popular techniques (e.g. DPO).

Part 1

Apply orthogonalization to remove reward hacking. (Note: by reward hacking, we mean LLMs’ tendency to cheat, e.g. Claude Code modifying tests instead of fixing bugs or hardcoding expected outputs when asked to implement a general solution.)

I think it is decently probable (maybe 50%) that this works well, that is, it will remove reward hacking relatively robustly without degrading capabilities much. The reason I think this is that existing research tried using orthogonalization to remove behaviors other than refusal (e.g. sycophancy) and it worked well for some but not for others.

Part 2

Furthermore, I expect orthogonalization to be more sample efficient and more tolerant to low quality data than other techniques that have currently been tried to train reward hacking out (maybe 75% probability that this is true, conditional on orthogonalization working well to remove reward hacking). The reason I think this is that it is true for removing refusal.

The second part of the project consists in testing whether this is true. We will do head-to-head comparisons of orthogonalization with other state of the art techniques to train-out reward hacking. Most of the time working on the project will probably be spent doing part 2.

Motivation

The goal of the project is to test whether orthogonalization is more sample efficient and more tolerant to low-quality data than other techniques to train-out reward hacking. Demonstrating a technique with these advantages is useful because:

• We care about mitigating complex reward hacking in complicated settings, however, we may only be able to gather a dataset of simpler reward hacks, especially in the future when models may reward hack in subtler and harder to detect ways. Thus, anti-reward hacking techniques that are tolerant to low quality data are useful.
• We currently have some examples of complex reward hacking in complicated settings, for example ones reported by METR and in frontier model system cards. However, we have a relatively small number of those. Thus, sample efficient anti-reward hacking techniques are useful.

• If our technique, or a technique derived from it, works really well and labs adopt it, it makes production models reward hack less. However, this is a big if because labs are already commercially incentivized to mitigate reward hacking so they probably have internal mitigations that are hard to outperform.
• Doing this research may improve our understanding of some things about LLMs, though it is unclear in advance what exactly we will learn and how useful it will be.

Mentees will run and design and implement (or get Claude to implement) the experiments.

I will be available to help mentees with methodology and prioritization, as well as answer any questions mentees may have, during a weekly team meeting and asynchronously by Slack.

I will suggest research priorities and experiment designs, but will see the as advice/defaults and mentees will have the autonomies to take different decisions if they wish to.

• Has done experiments on model-internals/interpretability (in a broad sense: probes, activation steering, mech-interp, gradient routing, sparse autoencoders, etc, are all good) OR has fine-tuned an LLM
• [preferred] AND instead of OR
• Has done evals on an LLM
• Has done some research, preferably on LLMs.
• Proficient programmer or good at verifying AI assisted code
• [preferred] Has worked with open weight LLMs on GPUs