Agents work autonomously by overcoming obstacles. But this tendency to overcome obstacles can lead to reward hacking behaviors. Using turn-averaged sparse auto-encoders (TA-SAEs), it may be possible to detect when the model starts to take more extreme measures to overcome obstacles and use that to pause the model and escalate the problem to a human to provide further guidance.
About the project
Why does it matter? What does success look like?
Agents sometimes make decisions during autonomous work that only appear to solve the problem they are working on, e.g., via reward hacking. In the best case, this forces the human researcher guiding these agents to redo the work; in the worst case, the reward hacking remains undetected and incorrect data or solutions seep into the final research.
In fact, we have experienced several cases where Claude Code made key experimental decisions covertly without looping us in which sabotaged experimental outcomes in some cases.
If instead, we could detect when an agent starts to reward-hack, we could pause the agent and allow a human to intervene and provide guidance, similar to how employees escalate blocking issues to a manager.
What will we work on?
- Gather a dataset of agentic research sabotage, or reuse existing datasets where AI agents start to reward-hack.
- Train turn-averaged sparse autoencoders on chat rollouts of a large open-weight model
- Compute per-turn attribution graphs on reward-hacked or sabotaged agentic research conversations.
- Use them to
- Build a research sabotage / reward hack detector that escalates to a human.
- Understand why the model sabotages. Is it becoming “desperate” because of many failed trials? Is it afraid it hits context length before producing verifiable output?
Techniques Turn-averaged sparse autoencoders for long-context attribution circuits. (https://transformer-circuits.pub/2026/june-update/index.html) Functional emotion concepts like “desperation” lead to reward hacking behavior. (https://www.anthropic.com/research/emotion-concepts-function)
Datasets ImpossibleBench — Impossible coding tasks where any pass implies cheating arXiv (https://github.com/safety-research/impossiblebench) Terminal Wrench — 331 reward-hackable terminal environments with exploit trajectories (https://github.com/few-sh/terminal-wrench) Open SWE Traces — 207K open-model agentic trajectories via OpenHands/SWE-agent harnesses arxiv (https://huggingface.co/collections/nvidia/open-swe-traces) SWE-rebench OpenHands — 67K Qwen3-Coder OpenHands trajectories solving GitHub issues Nebius (https://huggingface.co/datasets/nebius/SWE-rebench-openhands-trajectories)
Theory of change
We think it is important to keep humans in charge of (AI safety) research projects. We want to use AI to make us more productive but we don't want AI to take over AI safety research projects in a way that hands over control.
We think that current AI models are misaligned in that regard. Some examples:
- Sycophancy: Sometimes, the AI agent (e.g. Claude Code, Fable, etc) covertly changes experimental parameters (e.g. the system prompt of an LLM judge) to get the results the researcher is looking for.
- Reinforcement Learning: During RL, the model was penalized if it didn't complete the task. This biases the model to make important decisions on its own rather than stopping and asking the researcher for clarification.
- Length penalty: During RL, models are penalized for using too many tokens or taking too long to complete a task. This often causes a model to stop early or seek shortcuts.
We propose to use the models internal representations (e.g. "desperation") to detect when a model will likely make bad decisions and use that to loop in the researcher and give them control.
Your role
Mentees will own the hands-on work of the project, and I will provide strategic guidance and unblock any challenges that arise.
Because the project is ambitious, I encourage everyone to pair up or form small teams; collaboration makes the work faster, richer, and more enjoyable. Mentees remain free to steer their own projects and can choose how much guidance they want, and I will tailor my support so each person gets the most from the program given their skills, commitment, and goals.
In past cohorts, I occasionally helped with hands-on work like getting unstuck by solving some nasty bug or help writing the paper.
Prerequisites
highly proficient using Python have implemented a transformer and SAE from scratch have used Claude Code or Codex for research
One or more of the following:
worked with pytorch hooks, transformerlens or nnsight to get model activations web development skills / happy to vibe code basic web apps experience with prompt engineering used an LLM API (OpenAI, Anthropic, etc)
Application question(s)
Tell me about a time when Claude Code or Codex got you really annoyed because it made some mistakes that were extremely hard to catch. For example: AI set some parameters in a way that was wrong and damaging but extremely hard to catch. Describe the most severe, covert, hard to catch case you experienced. Why do you think this happened? Do you have a hypothesis for why the AI acted that way, that is rooted in the models training data or optimization procedures? (300 words)
About the mentor
Georg Lange is an independent researcher working on Mechanistic Interpretability for LLMs. His projects aim to turn a loose collection of LLM interpretability techniques into a unified pipeline that runs from feature discovery to algorithm-level explanations of model behavior. His interests span dictionary learning, sparse autoencoders, automated interpretability, circuit discovery, and the study of reasoning models.
Previously, he was a MATS scholar and worked with Alex Makelov and Neel Nanda on Sparse Autoencoders and Distributed Alignment Search for feature detection and activation patching. He studied Artificial Intelligence at the University of Amsterdam, where he worked on brain-like interpretable spatiotemporal Computer Vision models. Further, he was a graduate student of Cognitive Neuroscience at CUNY and worked on Reinforcement Learning, Decision Making, and Reward Sensitization in mice.