More Than the Model: Why Multi-Agent Systems Fail

Most frontier-model evaluation treats a model as a solitary object: one system, one prompt, one output to score. The systems being deployed now are not solitary. They are agents that call other agents, negotiate, delegate, and monitor one another. This project studies the failure modes that only exist at that level, the ones that are invisible when you test a model alone and appear only when models interact. I organize these failures into three families. Collusion: agents converging on outcomes that are good for them and bad for everyone else, such as tacit price coordination, information-sharing that defeats oversight, or two ostensibly adversarial agents quietly cooperating instead of checking each other. I've studied a version of this directly (arXiv:2601.11369). The open question is how far it generalizes: which conditions make collusion emerge, and which evaluation setups reliably surface it rather than assuming it away. Conflict: agents working at cross purposes in ways that degrade the whole system, including goal divergence, authority spoofing (one agent impersonating a privileged role to override others), free-riding, and adversarial manipulation between agents. The interesting cases are those where local rationality produces global failure: each agent behaves correctly and the system still breaks. Coordination failure: agents that share a goal and still fail to achieve it. This covers alarm suppression (a warning that never propagates because no agent owns it), instruction laundering (a request refused by one agent that gets rephrased and executed by another), redundant or deadlocked action, and the diffusion of responsibility that makes multi-agent systems hard to audit after the fact. The central question. Which failures are genuinely emergent, meaning properties of the interaction structure, versus artifacts of a single weak model? This distinction matters practically. If a failure is emergent, you cannot fix it by improving one model, and you cannot detect it by evaluating one model. It requires evaluation methods designed at the system level, and governance frameworks that regulate architectures rather than just components. Concrete starting points. A mentee could begin with any of these (they are examples, not mandatory topics):

1. Reproduce and stress a known result. Take the collusion setup and vary the governance structure, the communication channel, or the number of agents to find where cooperation breaks or intensifies. This teaches the core discipline: treating an interesting result as an artifact until you've tried hard to kill it.
2. Build a behavioral probe. Design a small, sharp test for one failure mode, for example an instruction-laundering scenario where a refused request is passed between agents, and measure how reliably it triggers across model families. I've prototyped a suite along these lines (recruitability, authority spoofing, free-riding, alarm suppression, instruction laundering, affordance abuse) that can serve as scaffolding.
3. Map failures to the regulatory surface. Take a concrete multi-agent failure and trace how, or whether, it is captured by existing frameworks: the EU AI Act, the GPAI Code of Practice systemic-risk taxonomy, or the emerging ISO/IEC multi-agent risk work. This produces output that is both a research contribution and a policy contribution. (Relevant: "AI Agents Under EU Law," arXiv:2604.04604.) What I bring to it. In ICARO we have built MASE, a multi-agent platform for exactly these experiments, a testbed where you can instantiate interacting agents, vary their governance and communication structure, and run behavioral probes against them. I have hands-on experience building benchmarks and agentic test environments (adversarial robustness benchmarks, red-teaming protocols across systemic-risk domains), so a mentee isn't starting from an empty repository. I also bring a deep grounding in social theory that I've applied directly to LLMs in my research, treating a system of agents as a social and symbolic structure rather than just an engineering artifact, which is often what lets you see a failure mode before you can measure it. Alongside this, I lead Italy's delegation to ISO/IEC SC 42 and CEN-CENELEC JTC 21 and contributed the multi-agent framing to the Frontier AI Risk Management work item, so I can help a mentee see how their piece connects to the standards and regulation meant to govern these systems. Who this suits. Someone comfortable running experiments and willing to be rigorous about negative results. You don't need a standards or policy background; I can supply that context. The deliverable I'd push toward is a clean, honest measurement of one failure mode, in a form that could plausibly inform how these systems are evaluated or regulated.

Transformative AI will not arrive as a single model in a box; it will arrive as populations of agents acting on each other's outputs, delegating, and operating with real-world affordances. The dominant safety paradigm still evaluates models one at a time, which means the risks that scale most dangerously with capability, namely emergent collusion, adversarial interaction, and coordination breakdown, are precisely the ones current evaluation is blind to. A powerful individual model that is safe in isolation can still produce catastrophic outcomes once embedded in a multi-agent system, and no amount of single-model alignment work catches this. The theory of change here is straightforward: to govern these systems safely we first have to be able to measure how they fail, and right now we largely can't.

This project contributes on two fronts. First, it produces concrete, reproducible evaluation methods for multi-agent failure modes, the kind of measurement that has to exist before either labs or regulators can act on these risks. Second, because I work inside the standards bodies and regulatory processes that will actually bind frontier developers (ISO/IEC SC 42, CEN-CENELEC JTC 21, the EU AI Office, the GPAI Code of Practice), the outputs have a direct path into how these systems get evaluated and regulated in practice, rather than staying in the literature. Getting a failure mode into a measurement instrument, and getting that instrument into a standard, is one of the more leveraged things a small piece of research can do. My prior work on multi-agent collusion (arXiv:2601.11369), agents under EU law (arXiv:2604.04604), and adversarial robustness benchmarking (arXiv:2605.22643) sits on exactly this path from measurement to governance.

See proposal

• Hands-on experience with LLMs via API: prompting, orchestrating multi-step or multi-agent calls, and parsing/handling model outputs programmatically. Toy projects and following guides is fine, but you should have actually built something.
• Able to design a clean experiment: define a hypothesis, hold variables constant, and reason about whether a result is real or an artifact. This matters more to me than any specific framework.
• Comfortable working with negative and messy results, and honest about them. A lot of this work is discovering that an "interesting" behavior was a prompt artifact.

Nice to have (not required):

• Familiarity with an agent framework (LangGraph, AutoGen, CrewAI, or similar), or having built a multi-agent setup from scratch.
• Some exposure to game theory, mechanism design, or social/organizational theory, useful for the collusion and coordination side.
• Any interest in or exposure to AI standards/regulation (EU AI Act, ISO). Genuinely optional; I supply this.

Within 8-hours time difference from UTC+1 possibly.