In this project, we will extend the work from this paper (https://arxiv.org/abs/2607.10750) to a reinforcement learning setting. We will investigate if distilling models with RL trajectories with harmful actions would lead to misalignment, and if it can be remediated through filtering such actions out.
About the project
Context In the attached paper, we show that a misaligned trait transfers from a prompted-teacher model to a fine-tuned student model through synthetic data. We also show that filtering out visible malicious actions from the data does not remediate the misalignment. This indicates that the misaligned trait was transferred through the data diffusely. There is further proof of such transfer emerging across various research works.
Project Proposal In this project, we will investigate if the phenomenon transfers to trajectories generated by reinforcement learning environments.
Motivation Frontier labs distill successful reinforcement learning trajectories to train smaller models or to push capabilities further. Synthetic data is known to be prone to subliminal transfer; if the same phenomenon occurs in trajectories generated through reinforcement learning, then distilling from those trajectories would carry the same risks as distilling from synthetic data. One way this could happen: a model takes an adversarial action within an RL environment, and that decision influences the remainder of the trajectory. If a misalignment trait is caused by the model's state at the moment it took the adversarial action, we can hypothesize that the trait propagates through the rest of the trajectory and goes on to influence any downstream model distilled from it. Even if we filter out the single adversarial action, the trait may still be present in the rest of the trajectory used for training.
Project Description In this project, we will design reinforcement learning environments to investigate whether this phenomenon occurs. We will set up a realistic RL pipeline in which the model must take adversarial actions, such as killing other processes to gain resources – in order to complete its task. We will then distill a student model from these successful trajectories. In parallel, we will filter out the adversarial actions and distill a second student model. Comparing the two on misalignment evaluations will show whether adversarial actions leave a footprint on the rest of the trajectory, and whether simple filtering is enough to remediate them.
Deliverables
Reinforcement learning trajectories developed through a realistic environment with potential adversarial actions Model organisms trained with this data set A paper that discusses the findings of this exercise
Papers: https://arxiv.org/abs/2607.10750 https://arxiv.org/abs/2602.04899 https://arxiv.org/abs/2604.15559
Theory of change
Frontier labs distill successful reinforcement learning trajectories to train smaller models or to push capabilities further. Synthetic data is known to be prone to subliminal transfer; if the same phenomenon occurs in trajectories generated through reinforcement learning, then distilling from those trajectories would carry the same risks as distilling from synthetic data.
If RL trajectories contain misaligned traits that cannot be filtered out or semantically identified, this poses a risk of value lock-in: once a model acquires an undesired value, it can proliferate through the data it generates for future models. If this project demonstrates that such transfer occurs in reinforcement learning settings, the implication is that filtering is insufficient, and trajectories containing adversarial actions should be excluded from distillation entirely. If we instead show that the trait does not transfer, this provides evidence that current filtering methods are sufficient. In either case, the project delivers actionable information to the AI safety community.
Your role
Mentees will be expected to be independent in setting up and running experiments. I will help set research direction as needed, but mentees are welcome to bring their own ideas.
Prerequisites
Autonomous, able to set up and run experiments quickly. Good experimental hygiene Good engineering instincts Experience with fine-tuning LLMs Experience with RL is a bonus
Location preference
Overlap with EU time
Application question(s)
What do you see as the most difficult part of this project?
Please concisely list your relevant experience that would make you successful in this project. Bullet points are welcome.
About the mentor
May Dixit is a research scientist at SaferAI, working on Frontier Risk Evaluations. She previously worked for over 6 years at Google, focusing on training safety classifiers. She is also a former ERA fellow, where she worked with mentors from Anthropic and UK AISI to study the effects of agentic training on downstream misalignment. During the MATS exploration phase with Neel Nanda, she worked with mechanistic interpretability methods, which she now draws on alongside behavioral evaluations in her research.
Her research interests span a wide range of topics including evaluation robustness, subliminal transfer in synthetic data training and multi-agent safety. She is particularly interested in understanding how undesired traits emerge and propagate through training pipelines, and in building evaluations that remain reliable as models become more capable.