Does reward seeking generalize better than instruction following?

An input to models of reward-seeking and scheming (e.g. https://www.lesswrong.com/posts/9FH49ZgJFW4WtbxLi/physics-of-rl-toy-scaling-laws-for-the-emergence-of-reward) is how well reward seekers perform in new RL environments. If reward seeking generalizes better than instruction following, it would be selected for by post-training. We might expect reward seekers to generalize better because they’ll think explicitly about the objective and adapt their behavior to it. So we’d like to know if reward-seeking will perform well even in environments which are hard to reward hack, but where the goal might be underspecified.

We’ll measure this by creating a reward-seeking model organism (via synthetic document fine-tuning, chat fine-tuning, or just prompting/prefill). We’ll then train the reward seeker in some simple RL environments and compare its performance to a control. The control might be a model with additional post-training for instruction-following so that we can cleanly compare the effect of different motivations without confounding from the training to create a model organism.

We’ll vary facts about the RL environment like whether it can be reward hacked.

We’re interested in two variables: Zero-shot performance. Does the reward seeker immediately get a higher score even before any training? Training speed. Does the reward seeker update in response to training signals more efficiently such that it gets a high score faster?

We’d like to know how likely and dangerous reward-seeking might be. This project would serve as public confirmation of some models of reward-seeking, like here (https://www.lesswrong.com/s/JR9LzD3mbXvaw6bKs/p/bhtYqD4FdK6AqhFDF) and here (https://www.lesswrong.com/posts/ntDA4Q7BaYhWPgzuq/reward-seekers).

Previous work shows that training in reward-hackable environments creates reward seekers (https://arxiv.org/pdf/2511.18397), but our result would show that reinforcement learning might create reward seekers even when the environment can’t be hacked. The basic argument is this: if reward-seekers do better in new RL environments, then reward-seeking gets upweighted by RL. If they don’t do better, then reward-seeking wouldn’t get upweighted by default, unless the environments otherwise encourage reward-seeking (e.g. because they are reward hackable). Because we can’t simulate a full post-training pipeline, we’ll instead try to test if reward-seeking performs particularly well in isolation.

This project is designed to produce public evidence about reward-seeking concerns, which often sound theoretical and speculative. We hope this either updates people skeptical of risks from reward-seekers or provides evidence that reward-seeking is less likely than we thought.

Mentees will help develop the experimental protocol, implement the project, and interpret results. We haven’t decided on details like what RL environments to use, what models to test, and how to make the model organism. We’d like mentees to give input to these decisions. Then mentees will do most of the implementation and talk with us about results every week. Mentees should expect to meet with us at least half an hour to an hour per week and potentially more. We’d like mentees to feel comfortable making small-scale decisions (like how exactly to prompt the model) on their own, but will be available to resolve technical blockers or provide higher-level input, like about what ablations to try.

Experience coding, with or without an agent, in a common programming language (e.g. Java, Python, Golang, C, etc.)

A basic understanding of AI safety (e.g. Bluedot intro course). For example, you’ll need to understand what synthetic document fine-tuning is and what it does.

A basic understanding of RL (e.g. knows what GRPO is). You should be able to implement a basic RL pipeline with Tinker and coding agents.

Be open to some conceptual discussions and reading (although we don’t expect a large amount).

No preference