Learning Interpretable Updates to LLMs

Post-training is key to attaining the best performance from large language models (LLMs). Procedures like direct preference optimization (DPO; Rafailov et al., 2023) and reinforcement learning with verifiable rewards (RLVR; Guo et al., 2025) allow us to take language models and turn them into useful conversational agents—and potentially to improve their ability to reason through challenging tasks. However, parametric updates can affect many aspects of a language model's behavior; this makes it difficult to understand what, exactly, has changed in its parameters. This has concerning implications for AI safety: what if these procedures only (un)learn superficial fixes while retaining potentially harmful information or biases? Interpretability typically relies on post-hoc analysis, which does not address the source of potentially new unsafe behaviors or biases at the root. Can we develop new post-training and unlearning procedures that are inherently interpretable? Can we use our understanding of what is being modified to monitor and control the post-training process?

Recent work has found that post-training and fine-tuning largely reweight existing concepts and mechanisms (Prakash et al., 2024; Venhoff et al., 2025); that is, few mechanisms in a post-trained model are truly novel relative to the base model. Thus, it seems highly likely that one could derive a procedure that directly learns updates over an interpretable concept space.

There are three main directions that seem promising given recent work. The first involves learning binary or continuous masks on interpretable spaces, such as sparse autoencoder features (cf. Ashuach et al., 2025, who developed a related procedure for unlearning, and Mueller_, Geiger_ et al., 2025, who learn binary masks over SAE features and PCA directions). A second direction could involve the use of crosscoders to perform "model diffing"—that is, learning a shared feature space for the base and posttrained model, and then interpreting the most causally relevant features to a behavior of interest in the posttrained model (cf. Bayazit et al., 2025). A final possibility could include learning low-rank updates to models in a semi-supervised fashion, where these directions are encouraged to align with known concepts—found, for example, via alignment search methods. This is conceptually similar to representation fine-tuning (ReFT; Wu_, Arora_ et al., 2024). These approaches vary in scope—masking SAE features is more surgical and easier to set up, while semi-supervised methods would suffer less from common pitfalls like feature splitting or low concept recall. A mentee could focus deeply on one approach or explore connections between them.

The above list is not exhaustive. If the mentee has their own promising idea, I am happy to provide them computational resources and mentorship while pursuing it.

TL;DR: Interpretable post-training methods could allow us to detect and fix potentially deceptive or fragile alignment. This project aims to develop such interpretable updates.

References: Tomer Ashuach et al. (2025). "CRISP: Persistent Concept Unlearning via Sparse Autoencoders." arXiv. https://arxiv.org/abs/2508.13650

Deniz Bayazit et al. (2025). "Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretraining." arXiv. https://arxiv.org/abs/2509.05291

Daya Guo et al. (2025). "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." arXiv. https://arxiv.org/abs/2501.12948

Aaron Mueller_, Atticus Geiger_ et al. (2025). "MIB: A Mechanistic Interpretability Benchmark." ICML. https://arxiv.org/abs/2504.13151

Nikhil Prakash et al. (2024). "Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking." ICLR. https://arxiv.org/abs/2402.14811

Rafael Rafailov et al. (2023). "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." NeurIPS. https://arxiv.org/abs/2305.18290

Constantin Venhoff et al. (2025). "Base Models Know How to Reason, Thinking Models Learn When." Mechanistic Interpretability Workshop at NeurIPS. https://arxiv.org/abs/2510.07364

Zhengxuan Wu_, Aryaman Arora_ et al. (2024). "ReFT: Representation Finetuning for Language Models." arXiv. https://arxiv.org/abs/2404.03592

Post-trained models can acquire biases or unsafe tendencies that the base model did not express as strongly. These new behaviors can be realized in ways that would be difficult to notice using behavioral evaluation or post-hoc interpretability. We would like to know precisely when and how this happens during training such that we can predict and lower the likelihood of a model behaving in undesirable ways after post-training. It could also help us assess how likely common jailbreaks are to work before and after posttraining or unlearning.

Mentees will be expected to propose and run experiments, and present the results from these experiments in weekly project meetings. Mentees will also be expected to lead the writing of a research paper to be submitted to a machine learning conference. Guidance will be regularly provided during all stages of the project, and the mentee is not necessarily expected to work alone (unless this is their preference).

• Proficiency with PyTorch.
• Can load a few-billion-parameter language model and efficiently run inference across many inputs (e.g., using Huggingface transformers).
• Can perform parametric updates to a language model, whether via supervised fine-tuning, low-rank adapters, or reinforcement learning.
• Can load, use, and ideally train external modules like probes or sparse autoencoders.

Mentees should be available for check-ins during some one-to-two hour slot within 9am - 6pm Eastern U.S. time.