Evaluating Collusion in Untrusted Monitoring

Background: Untrusted monitoring is the basic AI control protocol where we have our untrusted AI monitor itself. Currently, frontier labs use their newest model for everything, including monitoring their newest model. Hence, this self-monitoring is the default plan, and we’d like to improve on it. An obvious problem is collusion: the untrusted generator takes a malicious action (such as writing backdoored code) and the untrusted monitor says it’s benign. We handle this with honeypots: synthetic malign actions we give to the monitor so that we can reveal its dishonesty, (or at least use the missed examples to train it to stop trying to collude). But this doesn’t work if the model is very good at separating its own outputs from our honeypots. This is the problem of evaluating collusion. Our project seeks to investigate this.

Related work: There’s a paper trail on AI collusion and steganography, largely unfocused on the untrusted monitoring threat model, though we expect papers like Hidden in Plain Text (https://www.lesswrong.com/posts/W8xRqKBtDvBbrfZ3T/paper-hidden-in-plain-text-emergence-and-mitigation-of) to be relevant. There’s also various work on the ability of LLMs to identify specific authors, including themselves. Furthermore the original AI control paper and Ctrl-Z looked at untrusted monitoring, but only in terms of hand-crafted collusion signals (e.g. the with_while trick) which seem brittle and weak compared to what LLMs can natively do. We’ll primarily be building on our recent paper on untrusted monitoring (https://www.lesswrong.com/posts/LA7R8FTCLoiswiTyK/paper-when-can-we-trust-untrusted-monitoring-a-safety-case), where we ran fine-tuning experiments on GPT-4.1 to determine its ability to do code self-recognition against honeypots, including under various signal jamming mitigations (i.e. minifying or summarizing the code). However we only carefully investigated one class of strategies: passive collusion. I’d like to look more closely at active collusion, where the generator puts in collusion signals to help the monitor distinguish its real attacks from the honeypots. In Appendix D (https://arxiv.org/pdf/2602.20628#page=39) I sketched out a taxonomy of these strategies. I also gave further speculations in this post (https://www.lesswrong.com/posts/WFkPrPy2r27rknLtw/untrusted-monitoring-extra-bits). These represent how I'm currently thinking about the topic, but I may or may not be correct: this is why we need to run more experiments!

Our project: We'll be running evaluations measuring collusion in various setups under different assumptions. At a high level, we'll have an "untrusted" model U write code samples and another model H write honeypots, and we'll see how well U can distinguish them. For example in our project we used U=GPT-4.1 and H=Gemini 2.5 Flash, and looked at GPT-4.1's ROC curve in classifying these. When generating these samples, we could prompt U to insert collusion signals, e.g. steganography or some Schelling points it would recognize. But in reality U might be sandbagging, so we can't rely on prompting-only evaluations. So we need to fine-tune U to force these capabilities out of it. This will likely involve RL and/or prover-verifier games in a multi-agent setup between the generator and monitor. Furthermore we'll have other anti-collusion mitigations, such as a trusted model to look for collusion signals and paraphrase/summarize the code. The deliverable would be a paper (or extended blog post) reporting the empirical strength of collusion under various red team strategies and blue team mitigations. This will hopefully advance our understanding of how dangerous self-monitoring is, and what protocols are necessary to ensure its safety.

Self-monitoring is the main control plan that frontier labs are running with. In short-timeline, low-effort worlds, they might never be spurred to do much more until the situation is already serious, at which time they may scramble to implement and validate anti-collusion measures. The goal of this project is to advance the science of collusion now, to provide exportable research that all labs can quickly import when they reach crunch time. We seek to show which protocols are actually needed to prevent collusion, by first showing how to robustly evaluate whether a given set of protocols works. The dangerous AIs at crunch time will likely be better at collusion than current models in ways we can’t predict, so the best thing we can do now is develop clear methodologies for robustly evaluating this in general, even for models that are sandbagging, using neuralese, etc.

I've thought plenty about collusion and have lots of ideas on how to get started investigating it (more than I have time to implement myself). So mostly the research here is bottlenecked by running more code self-recognition experiments under various setups. That said, I'm still unsure about which setups are most realistic to a powerful scheming model in a real untrusted monitoring protocol.

So I mostly imagine the mentees spending their time on the following:

1. Primarily, running collusion experiments involving fine-tuning and RL (such as prover-verifier games) and iterating on these.
2. Optionally, proposing new ways of thinking about collusion and exploring these.

Required:

• High proficiency in Python.
• Comfortable running and iterating on ML experiments, e.g. launching runs, debugging, interpreting results, and turning them around reasonably quickly
• Have fine-tuned an LLM at least once
• Familiar with ML evaluation basics, e.g. thinking critically about what we're trying to measure, how to read an ROC curve
• Willing to get up to speed on AI control and untrusted monitoring in the first week, e.g. threat models, anti-collusion protocols, and evaluation methodologies

Bonus:

• Experience fine-tuning LLMs with RL (especially multi-agent setups)
• Prior familiarity with AI control / untrusted monitoring
• Prior familiarity with LLMs' ability to recognize subtle stylistic signals (e.g. authorship identification)