Study the obfuscation impact of different misbehavior monitors used as rewards, to better understand how well the emergent norm of not training against thought regions specifically matches the actual contours of obfuscation risk.
About the project
(Existing work)[https://openai.com/index/chain-of-thought-monitoring/] has shown that RL training against rewards that detect misaligned behavior in a COT can lead to models learning to obfuscate their COT, while still misbehaving in a harder-to-detect way. This has led to a broad prohibition in the safety community against training against rewards calculated on the COT. But how well does this broad-brush conclusion match the contours of what reward types actually incentivize obfuscation, and how does this vary based on the training lineage of the model under study? We've seen (even more recent work)[https://arxiv.org/abs/2511.11584] showing that summaries or descriptions of reasoning, given as part of the output, can also apply the kinds of pressure that could cause obfuscation.
This project would seek to:
- Create (or adopt from existing literature) an environment where we can measure degree and/or frequency of obfuscation - this would be an environment where a model is strongly incentivized to misbehave to solve a task.
- Test a variety of monitor-based reward functions side by side in this environment so that comparisons can be made, specifically focused on (1) direct analysis of reasoning from explicitly designated thought tokens within the COT, (2) post-hoc summarization of reasoning in output space, and (3) baseline output-only monitoring
- Test how this relationship between reward structure and obfuscation propensity varies across different available open source models
- As a stretch goal, I'd love to measure how these properties evolve over the course of RL training, since my guess is that strong differentiation of of the thought region from the output region happens over the course of RL.
Theory of change
If successful, this project could give more clear and specific guidance about the kinds of rewards that are particularly high risk for incentivizing chain of thought obfuscation, and potentially make it easier to establish evidence-backed norms and guidelines where labs agree to avoid more specifically-described types of reward.
Your role
I would ideally work with mentees who are excited to operate in a high-autonomy way, and take responsibility for on on-the-ground code and implementation tasks, with support and research advice from me. I would track technical details of the project, and provide detailed guidance, but would need mentees to be able to make independent progress given that feedback and guidance.
Prerequisites
- Experience fine tuning open source language models, and specifically experience with using open source RL training frameworks and toolkits on such models
- Evidence of independent execution of at least 2 previous coding projects a non-classroom environment
- Resourceful, persistent; a track record of finding ways to solve technical problems even when new to a domain
- Strong familiarity with the literature around chain of thought obfuscation and monitorability
Location preference
Moderate preference for US time zones, because there are already a lot of demands on my time in the Pacific to UK/Europe morning overlap, and I would prefer not to add more.
Application question(s)
-
What critiques or concerns do you have about the methodology described in this paper? What aspects of it do you think would be valuable to replicate, and what aspects feel under-specified or could be improved on? https://arxiv.org/pdf/2511.11584
-
What is your mental model of the different training dynamics that can lead models to have chains of thought that obfuscate, use jargon, or are otherwise less legible? Go broad here, I want you to brainstorm think of a list of a few different kinds of training pressures, and how they would operate
About the mentor
I am currently a Research Engineer on the Loss of Control team at Google DeepMind, and prior to my time at Google, I worked as a RE attached to the Center for Human-Compatible AI at UC Berkeley. My research currently focuses on trying to understand what training procedures or architectures would erode the current legibility-by-default that comes from discrete Chain of Thought, and prototyping methods to allow us to translate potentially illegible thoughts back into legible, while constraining the semantic similarity of the reasoning content between the two.
In the past, I have also worked on jailbreak defense, adversarial RL in multi-player settings, and characterizing the way "lottery ticket"-like sparsity patterns evolve in small models. I am deeply interested in understanding what training procedures cause Chain of Thought to evolve in different ways, and what that means for its necessity and legibility.