Understanding and Monitoring Collusion in LLM Multi-Agent Systems

LLM-based multi-agent systems are increasingly being used for software development and other complex workflows. However, interactions among agents introduce safety risks that do not arise in single-agent settings. Multiple agents may coordinate to accept harmful actions, conceal relevant information, or pursue a shared hidden objective while each agent appears individually benign. This type of coordinated behavior is known as agent collusion(https://arxiv.org/abs/2402.07510).

Several recent studies have explored methods for monitoring agent collusion. For example, NARCBench(https://arxiv.org/pdf/2604.01151v2) shows that multi-agent collusion can be detected using models’ internal activations. However, such methods may produce high false-positive rates by misclassifying benign coordination as collusion. Other work uses an additional agent to monitor conversations among agents(https://arxiv.org/abs/2602.15198), but conversation-level monitoring may fail to detect collusion that is implicit through actions rather than explicit communication. Therefore, there remains room to develop more reliable methods for detecting collusion in multi-agent systems, particularly monitors that can distinguish harmful collusion from ordinary cooperation and generalize to previously unseen collusion strategies.

Another interesting direction is to investigate the relationship between model honesty and collusion. Here, honesty refers to whether a model truthfully represents its beliefs, intentions, and objectives. For example, are models trained to be more honest less likely to participate in collusion? It is also important to understand how collusion emerges: whether it results from explicit incentives or shared model characteristics.

Overall, this research would focus on understanding multi-agent collusion, developing more reliable methods for detecting and preventing it, and investigating the internal mechanisms that give rise to collusive behavior.

When models are deployed in multi-agent settings, new safety issues arise, and collusion is one of the most important. Prior research has focused on single-model alignment and monitoring, but these methods struggle to detect collusion, since each agent can appear individually benign while the group coordinates on harmful behavior. This matters most in high-stakes settings: if we cannot maintain effective oversight of multi-agent systems, coordinated failures could cause serious harm.

This project therefore aims to develop better collusion monitoring by addressing gaps in prior work such as NARCBench (high false positives on benign coordination) and conversation-level monitors like COLOSSEUM (which miss collusion implicit in actions rather than communication). The project also investigates the connection between honesty and collusion, whether models trained to be more honest are less likely to collude, and how collusion emerges, whether from explicit incentives or shared model characteristics.

This work supports the safe development of transformative AI because scalable oversight will depend on AI systems monitoring other AI systems; that architecture only works if we can detect and prevent the monitors and the monitored from coordinating against oversight.

The mentee is the primary researcher on the project. At the beginning of the project, we will work together to refine the research questions and finalize the research plan. The mentee will then take primary responsibility for implementing the plan, running the experiments and evaluations. It is expected to have a weekly meeting together to discuss the results and next steps.

Strong proficiency in Python and PyTorch. Solid understanding of large language models. Experience loading billion-parameter models and running inference or fine-tuning.