AI agents are increasingly used to negotiate and act on behalf of people and organizations, but there is little empirical work measuring how interactions between such agents fail. This project builds an open testbed for bargaining and crisis interactions between frontier-model agents and runs initial experiments on bargaining failure, escalation dynamics, and commitment mechanisms.
About the project
Bargaining theory suggests that delegation changes strategic interaction: information flows differently, new commitments become possible, and agents can fail in ways their principals would not. There is currently little empirical work testing these dynamics with frontier models, even though interaction-level failures (escalation spirals, bargaining breakdown, exploitation of weaker counterparts) can occur without any individual agent being misaligned in isolation.
The project has an infrastructure component and an experimental component. The testbed consists of a scenario schema (payoffs, private information, communication channels, escalation options, and monitoring hooks), a harness built on UK AISI's Inspect framework, adapters for frontier and open-weight models, and metrics for bargaining efficiency, escalation propensity, deception, and surplus division.
Three experiment families run on this infrastructure. The first examines bargaining failure, varying information structure and incentives to misrepresent, with Fearon's rationalist framework (Fearon 1995, https://web.stanford.edu/group/fearon-research/cgi-bin/wordpress/wp-content/uploads/2013/10/Rationalist-Explanations-for-War.pdf) as a theoretical anchor, to identify the conditions under which agents fail to reach efficient agreements. The second examines escalation: existing wargaming studies measure the escalation propensity of individual models responding to scenario prompts (e.g., Rivera et al. 2024, https://arxiv.org/abs/2401.03408), and this project instead studies escalation as a property of the interaction between two agents with feedback between their decisions. The third implements commitment mechanisms from game theory, program equilibrium (Tennenholtz 2004, https://www.sciencedirect.com/science/article/abs/pii/S0899825604000314) and safe Pareto improvements for delegated game playing (Oesterheld and Conitzer 2021, https://www.cs.cmu.edu/~conitzer/safeAAMAS21.pdf), which become implementable when negotiating agents are inspectable software; we are not aware of prior implementations with frontier-model delegates. A possible extension for a strong team is collusion under monitoring in negotiation settings.
Theory of change
Escalation, collusion, and commitment failure between agents acting for different principals are identified in the multi-agent safety and cooperative AI literatures as understudied relative to single-agent alignment, and they can produce severely harmful outcomes without any individual system being misaligned. The testbed provides measurement infrastructure for studying these failures. The commitment-device experiments test a mechanism that these agendas identify as a candidate stabilizer but that has not been implemented with current models. Results are intended to inform both technical follow-up work and policy analysis of the conditions under which delegated negotiation can be relied on.
Your role
Mentees build and own components of the project: parts of the schema and harness, or an experiment family end to end (design, implementation, sweeps, analysis, and write-up). I set the direction and review designs and code; the team meets weekly, and I provide asynchronous feedback within a day or two. Mentees who lead an experiment family would be first authors on its write-up.
Prerequisites
High proficiency in Python. Experience building with LLM APIs (agents, scaffolds, or evaluation harnesses; personal projects are fine). Comfort reading empirical ML papers. For the commitment-device experiments, basic game theory (familiarity with Nash equilibrium and the role of commitment in games). Experience with UK AISI's Inspect framework is helpful but not required.
Location preference
No geographical preference; weekly meetings between 11:00 and 18:00 UTC.
Application question(s)
You may use AI assistance on questions 2-3, but state how you used it; the explanation doesn't count toward word limits.
- Link to a code repository you wrote (required).
- (15 min) Two LLM agents with private valuations negotiate a split of a resource. What would you log to distinguish "the agent misrepresented its valuation strategically" from "the agent made an error about its valuation"? (200 words)
- (10 min) Suppose a frontier model negotiates twenty rounds against a much weaker open-weight model over a divisible resource with private valuations. What outcome would you expect, and why? Responses are evaluated on reasoning rather than on the specific prediction. (150 words)
About the mentor
Amritanshu Prasad is an AI safety researcher working on strategic interaction under AI delegation: how bargaining, crises, and agreements change when AI agents act on behalf of states and companies. He is a member of the working group on international AI governance at Uppsala University's Alva Myrdal Centre for Nuclear Disarmament and studies the strategic parameters of historical arms-control agreements in collaboration with the Oxford Martin AI Governance Initiative.
Previously, he was a fellow at Pivotal Research, where he developed coordinated vulnerability disclosure frameworks for AI systems with Robert Trager; co-authored a scheming-propensity evaluation paper at LASR Labs under the supervision of David Lindner (Google DeepMind); contributed to UK AISI's ControlArena; and worked on METR's HCAST benchmark.