Evaluate how much the effects of RL safety interventions arise from changing which trajectories the model explores, rather than changing other aspects of the training dynamics.
About the project
The project will reproduce a few interventions such as SDF + RL, inoculation prompting + RL, recontextualization + RL, or Open Character fine-tuning + RL. We will estimate how much of each intervention’s effect is mediated by changes to exploration.
References:
- Azarbal et al., “Recontextualization Mitigates Specification Gaming without Modifying the Specification”. Separates the prompts used for trajectory generation from those used for training, producing large differences in reward hacking and directly motivating an exploration-focused decomposition.
- MacDiarmid et al., “Natural emergent misalignment from reward hacking in production RL”. Finds that a one-line inoculation prompt during RL reduces misaligned generalization by 75–90% despite reward-hacking rates above 99%, motivating investigation of the responsible training dynamics.
Theory of change
Exploration determines which behaviors receive optimization pressure during RL. Identifying its causal contribution could make safety interventions more predictable and reveal inexpensive ways to steer post-training away from dangerous behaviors.
Your role
Mentees will have autonomy in how they approach the work and can rely on mentors for help with defining ideas, experiment design, prioritization, interpreting results, conceptual de-risking, and related work. Mentees will run experiments independently and report results, but I'll keep a fast feedback loop through Slack. They should communicate often.
Prerequisites
- Proficient in using Python or another programming language
- Experience evaluating LLMs
- Able to work independently and asynchronously
- [optional but encouraged] Experience training or finetuning LLMs
- [optional but encouraged] Proficient in designing, debugging, and analyzing experiments
Application question(s)
- Describe a technical achievement you are proud of. What did you personally implement, what was the hardest technical obstacle, and how did you determine whether the result worked? Include links when available. (200 words maximum)
- Critique “Recontextualization Mitigates Specification Gaming without Modifying the Specification”. Identify one conclusion you find well-supported and one you find weakly supported, referring to specific evidence from the paper. (200 words maximum)
- Assume you have access to the code and checkpoints for one of the candidate RL interventions. What first experiment would you run to determine how much of its effect is caused by changing exploration? State the components and expectations. (300 words maximum)
- How would you define and measure exploration in language-model RL? Propose two metrics, discuss their limitations, and explain which you would use first. (200 words maximum)
About the mentor
I have been working at the Center on Long-Term Risk on S-risk reduction for the last 5 years, initially as a research engineer and increasingly as a researcher. Previously, I worked on object detection in satellite imagery and, before that, on nanotechnology for energy storage.
For CLR, I mostly worked on private research around multi-agent dynamics, MARL, cooperation & conflict for the first two years. And in the last two years, on LLM evaluation, model organisms, and mitigating misgeneralization.
I contributed to studying inoculation prompting through exploring the idea at the start of 2025 and co-mentoring during this summer Daniel Tan, who led the work for the paper "Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time".