Towards exhaustive model diffing

Interpreting language models is difficult. Models are nonlinear in many places and memorize a lot of data, so they are best seen as monoliths that are tuned to perform many tasks we will never even be able to define.

However, there may still be some specific aspects of the behavior they perform that have simple explanations. One reason to expect this is that some behaviors are learned from only a few samples and likely don't produce that big of an update to model weights. This also may happen with a lot of samples if the behavior being encoded is simple conditional on the model having infrastructure to support it; e.g., the refusal behavior learned in finetuning in older models was probably not very complicated because it leaned on pretraining data for detecting what to refuse and also pulled in some aspects of the style of the response. That is, because the data specifying the behavior is simple, we expect the circuit to be simple, conditional on other tasks being learned well.

Such behaviors may be representable as a low-rank contribution to model weights because it is the sum of a few correlated rank-1 gradients and updates; we would expect parameter decomposition methods to recover them, and tools to interpret such model weight diffs would be the same ones we would use for interpreting parameter components. We are looking for diffs which correspond to how the model actually learned to perform the task; realism seems achievable because there are instances of the model learning something with just a few gradient updates and preserving the behavior through training. We are not biased about which tools we would use and would follow the level of description which makes the most sense and interpret to the extent generalizable structure actually exists.

Past work -- https://arxiv.org/abs/2506.11618, https://arxiv.org/abs/2507.08218, https://arxiv.org/abs/2510.13900, https://arxiv.org/abs/2606.00995 -- explained how some simple fine-tuning objectives basically work by injecting an unconditional steering vector that shifts model behavior on all prompts. They study an instance of the type of behavior we're looking for, and one of the directions we may pursue in the project is extending their characterizations of the steering vectors that are learned so we are 1) able to create such steering vectors by hand or with minimal finetuning 2) explain which components they affect in the base model and how.

Although steering vectors are pretty expressive and interpreting them is worthwhile, not all behavior in e.g. SFT or RL post-training can be described through them, and steering vectors may not be the most faithful explanation for everything. One type of behavior which is not as easily described through steering vectors is conditional behavior gated on some condition.

Backdoored models and non-leaking SDF are instances of this: the model cannot learn the behavior specified by the dataset without breaking capabilities on unrelated data, and multiple behaviors can be learned without interference. Based on some preliminary results, we believe those two finetuning methods specifically work in a somewhat similar way: they inject steering vectors, but conditional on some direction in the residual stream or hidden activations, and this direction can be interpreted straightforwardly through maximum activations (https://arxiv.org/abs/2511.06739).

This project will involve developing and publishing these results, improving our techniques for understanding the conditions the behaviors activate on, the behavior they're steering towards, and our ability to detect them under interference from unrelated signal or other tasks. The north star of this direction is to use many unrelated diffs to characterize the mechanisms by which models store knowledge more generally.

1. Model diffing techniques that can be applied internally to diagnose weird behaviors or backdoors
2. Learning about how gradient descent teaches models simple behaviors (the goal of this project) seems useful for basic science and understanding of language models. Helpful in long timeline worlds Overall, I will admit this project is pretty risky, but at least some directions are validated and will be useful (interpreting SDF and backdoored models)

This is my main research direction, so I will be fairly involved and will likely steer the direction. I expect mentees to understand the motivation behind this project pretty well and be able to propose and iterate on specific direction well on their own I think there are a lot of low-hanging fruit in this research direction, so mentees will probably be able to find well-motivated ideas fairly easily. However, I will check in and debug the progress to make sure we focus on finding directions which actually will produce something useful, clean and publishable (see must-have prerequisites).

I am mostly selecting for research taste, i.e. ability to tell good research / methodology from bad and stay focused on promising directions. I.e.:

• https://www.lesswrong.com/s/5GT3yoYM9gRmMEKqL/p/hjMy4ZxS5ogA9cTYK
• https://www.alignmentforum.org/posts/eJGptPbbFPZGLpjsp/highly-opinionated-advice-on-how-to-write-ml-papers
• https://nicholas.carlini.com/writing/2026/how-to-win-a-best-paper-award.html I want to see projects, drafts or non-technical writing which demonstrate a good grasp on this.

Experience doing interpretability or other taste-heavy research is helpful, of course. Lack of technical experience is fine, but it's highly recommended to be familiar with (or be able to learn before the project starts):

• transformer architecture, what's special about MLP layers
• neural network optimization and typical finetuning setups
• LoRAs
• papers on model diffing (including the ones linked in project description)
• metamodels (activation oracles, NLAs etc)
• persona selection model adjacent papers (persona vectors, assistant axis, emotion vectors) Knowledge of diffusion models, parameter decomposition and second-order methods (e.g. knowing what a Gauss-Newton matrix is) could also be helpful