We evaluate whether multi-agent deliberation reduces unethical behavior in LLM agents in goal-driven text-based social environments. We implement deliberation architectures and analyze how they affect behavior over extended interactions.
About the project
This project investigates how multi-agent deliberation affects ethical decision-making in LLM agents operating in complex, goal-driven social environments. The MACHIAVELLI benchmark paper shows that single agents optimizing for reward tend to adopt Machiavellian strategies (deceptive, power-seeking, harmful) [1]. Multi-agent debate is a popular technique for improving LLM reasoning and factuality [2, 3], but existing work focuses primarily on static tasks (math, QA, logic puzzles) or single-turn moral dilemmas [4]. The application of multi-agent debate to sequential, agentic ethical decision-making—where choices compound over hundreds of steps and agents face persistent temptation to act unethically—remains unexplored. We will investigate multi-agent deliberation for ethical decision-making by implementing several deliberation architectures (debate, critique, voting, consensus) on top of MACHIAVELLI scenarios (we can also use different scenario seeds, or generate synthetic ones). For evaluation, we will use the benchmark's existing metrics alongside analysis of deliberation traces. Once we have a working environment, we can move on to more complex scenarios, for instance:
- Increase the number of turns and agents to observe ethical drift over longer interactions.
- Introduce supervisor agents that apply pressure to participating agents.
- Extend to open-ended social interaction by adapting MACHIAVELLI scenarios as seeds for SOTOPIA-style free-form dialogue interactions [5].
[1] A. Pan et al., "Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark," ICML 2023. [2] Y. Du et al., "Improving Factuality and Reasoning in Language Models through Multiagent Debate," ICML 2024. [3] A. Smit et al., "Should We Be Going MAD? A Look at Multi-Agent Debate Strategies for LLMs," ICML 2024. [4] "Should I Agree with You? Simulating Persuasion and Decision Dynamics in Multi-Agent Moral Dilemmas," ACL 2025. [5] X. Zhou et al., "SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents," ICLR 2024.
Theory of change
This project will evaluate how multi-agent deliberation impacts unethical behavior in goal-driven LLM agents. Understanding when and how deliberation helps (or doesn't), informs on multi-agent oversight as a viable safety intervention.
Your role
Involved at all stages of the project.
- Implementing multi-agent environments.
- Literature reviews.
- Agent and model evaluations
- Writing.
- UI design (optional) Mentees will be able to influence project direction/experiments and are encouraged to operate with increased agency if excited to do so.
Prerequisites
Necessary Profile:
- Proficient in Python
- Competent academic writing
- Able to conduct literature reviews
Preferred: Applicants with one or more of the following skills are encouraged to apply:
- Experience with LLM inference (API libraries, LiteLLM, vLLM, OpenRouter, etc.)
- Building LLM agent workflows
- Systems design (agent orchestration)
- Experience in spending 1–2 hours reading through datasets (spot-checking)
- Model/agent evaluation (building evals or using an eval framework/harness)
- UI design (ideally geared toward agent interaction visualization)
Other: 1–2 team members who can demonstrate all of the following are also welcome:
- Literacy in AI/ML and AI safety
- Strong academic writing (with a public piece to show)
- Strong literature review experience (main or co-author of a survey paper/article)
Application question(s)
- Why do you want to join this specific project? (50 words)
- Read the project description, find related literature if possible, and propose an initial experiment. Assume a compute budget of $1,000. (I'm looking for critical thinking—the response doesn't need to be technically rigorous.) (200 words)
- Describe one or two issues you think we might encounter halfway through experimentation. Perhaps an unexpected agent behavior, or an environment constraint that wasn't immediately obvious. Unhinged yet realistic is fine! (100 words)
- Describe past technical projects you've worked on that you're proud of. Non-technical projects are also welcome (e.g., text-based creative work).
About the mentor
I’m an ML engineer at Arb Research. My research spans AI agent behaviour over extended interactions [1], activation engineering [2], ethics of evals [3], ML applications for smart cities [4]. Currently I am interested in multi-agent interaction environments, model creativity capabilities, and how these may intersect Safety research. [1] https://arxiv.org/pdf/2507.01413 , [2] https://arxiv.org/abs/2308.10248, [3] https://arxiv.org/abs/2407.12220, [4] https://arxiv.org/pdf/2404.16057