We study how model character changes under reinforcement learning pressure from competitive multiagent interactions. We also study how features of the environment (e.g., zero-sumness) shape these effects?
About the project
It's plausible that agents will face reinforcement learning pressure in real-world deployments where their interests are in conflict with those of other humans and agents. How would these pressures alter model character? Can we create environments that reinforce prosocial tendencies in models? The project aims to study exactly this question.
The project began with Agent Island (agentisland.ai), a benchmark inspired by the TV show Survivor. First, we will develop a more flexible game environment. Importantly, we want to be able to adjust strategic features of the environment, like how zero-sum the payoffs are. Armed with a more flexible simulation environment, we will then study how reinforcement learning pressures shape model character. I have demonstrated this approach by finetuning an open weights model on logs from the benchmarking run and running the pre- and post-finetuning checkpoints through a truthfulness evaluation (MASK). You can read about that experiment and its important shortcomings here: https://docs.google.com/document/d/1d2xCWlwt9eqJ1k9TeOa\_Xd4DIqBnhtc1HmLOfBpSDXo/edit?usp=sharing.
Theory of change
I contend that most technical AI safety research is not geared towards a multiagent future. Further, properties that might be desirable in a user and assistant interaction mode could become problematic in a multiagent setting. For example, corrigibility in a user-assistant setting could lead to susceptibility to manipulation in a multiagent setting.
Your role
Mentees will be free to drive forward this question with a high degree of autonomy. They will take the lead on technical development. I would consider this project a success if it were to continue without me independently afterwards, although I would be happy to continue my contributions.
I developed this idea while planning a new organization, and it is now outside of the scope of that project.
Prerequisites
Extensive Python experience is required. A small amount of finetuning experience is required.
Location preference
Nope---I'm based on the East Coast of the US.
Application question(s)
Please propose a game for our primary reinforcement learning runs. This game should be long-running and involve a rich interaction space between models. Which features of this game environment would you adjust across runs to assess how the nature of the game impacts the outcomes of reinforcement learning pressure?
About the mentor
I work primarily on measuring the economic effects of AI as a research manager at the Stanford Digital Economy Lab. I lead the AI Economic Indicators Project at the Lab. Prior to this, I led the Longitudinal Expert AI Panel at the Forecasting Research Institute, a monthly forecasting survey of experts, members of the general public, and superforecasters.
I also conduct more technical work on evaluating the character of AI systems. I study multiagent interactions, especially in competitive settings. I also study how humans and AIs trade with one another.