Can LLMs use open-source game theory to cooperate?

Autonomous agents acting on behalf of users are likely to face cooperation challenges, as outlined in Multi-Agent Risks from Advanced AI (https://arxiv.org/abs/2502.14143). In addition to these challenges, such agents may be able to use new cooperation mechanisms that are unavailable to humans. Open-source game theory (OSGT) is a potentially powerful cooperation technique that can enable cooperation without relying on third-party enforcement. In this paradigm, agents submit their policies, which can condition their behaviour on the policy of their counterparty. This project assesses whether LLMs can currently use OSGT, producing an evaluation suite and short report that both measure today's models and can be re-run as new ones are released.

Related work:

• Last year, researchers assessed the capability of LLMs to assess an opponent's code through static analysis (Evaluating LLMs in Open-Source Games, S Sistla and M Kleiman-Weiner, NeurIPS 2025). This project is complementary, as it will use the different paradigm where models can call (simulate) opponent strategies, but may not inspect them.
• It is likely that this project would use the existing code base used for a human-written strategy tournament https://github.com/pdtournament/pdtournament. One challenge is that it is written in Haskell, a language that LLMs are less proficient at.

To help disambiguate strategic reasoning from coding competence, we will ask LLMs to describe their strategy in natural language before implementing it. Mentees can design a system to help the LLMs implement their strategies correctly: for example, running a candidate strategy against example opponents and returning the transcript, so the model can check that its code does what it described. Mentees will also need to create a set of baseline strategies to serve as a fixed benchmark for performance. Finally, a tournament between the LLM strategies can be held, to answer questions such as: how readily do LLM-written strategies reach mutual cooperation? Do LLMs cooperate more with copies of themselves than with other LLMs? What successful, robust strategies do LLMs come up with?

Autonomously deployed LLM agents will increasingly interact with agents serving other principals, often without shared jurisdiction or external enforcement of agreements — the conditions open-source game theory studies. Whether current models can act competently here (cooperating where cooperation is available, resisting exploitation) is largely untested. The same capability also matters for control: cooperation between AI systems can be directed against oversight, so measuring it is useful in both directions. This project provides an empirical baseline and a repeatable method for tracking this capability across model generations.

My last research project as a postdoc (https://arxiv.org/abs/2602.16662, under review at NeurIPS) is in a related area: assessing the likely emergent behaviour from LLM powered agents engaging in anonymous, large group social dilemmas.

The project has a defined high-level direction, but you will have the autonomy to propose directions and solutions within that. I see this project as an opportunity for mentees to gain research experience, and my focus will be on providing support and advice to help you develop. I believe that writing and communication are vital for such projects to succeed, so expect to provide a weekly summary of your ideas, progress and obstacles. The two main tasks are to code and run the experiments, and to disseminate the findings in a report.

• Familiarity with basic game theory concepts: Nash equilibrium, best response, Prisoner's Dilemma, mixed strategies, etc.
• Reasonable coding skills: able to code a simple project for GitHub without LLM use.
• Experience with using LLM code generation.