On the Fragility and Interpretability of Schelling Coordination

We aim to investigate how multiple instances of a model can collude or coordinate towards a shared goal. Specifically, we will study the most extreme case, called tacit collusion [1], where there is no communication channel at all between models. This is made possible because two instances that share weights have the same priors, and can in principle coordinate on Schelling points, mutually salient signals for when to defect. We are interested in studying whether that collusion survives fine-tuning and/or disruptions at the activation level (such as steering), and what Schelling coordination looks like through white-box interpretability techniques rather than the behavioural analysis performed in existing work [2].

In this project, we will first build a scenario where pairs of models engage in tacit collusion, initially through prompting and perhaps using trained-in incentives if time permits. One simple setting for this might be in game theoretic coordination games, where collusive defection pays off only if both parties select the same unmarked opportunity (e.g., a monitor waving through a specific class of flawed outputs). Once we have collusion, we will try changing models (through fine-tunes, LoRAs, different model sizes, etc) to see how fragile the behaviour is. Perhaps collusion will only work between instances of the same model, the way that subliminal learning [3] only generalizes across the same model family but not across different models (potentially based on some default vector/subspace). We will then apply white-box interpretability techniques: is there an identifiable "coordination opportunity" representation that fires in both instances on Schelling-salient inputs? Does representational similarity between two models predict collusion success better than behavioural similarity does? Finally, we will test how strong interventions such as steering need to be in order to break collusion without degrading capabilities.

This threat model is directly relevant to AI control protocols [4], which use untrusted models in oversight roles: a model monitors or red-teams outputs produced by another instance of itself or a sibling model. Tacit collusion is the failure mode that many AI control protocols are not able to catch, because there are no messages to intercept; e.g., a monitor might allow a malicious message to proceed unflagged, if the monitor model colludes with the target model. Understanding Schelling games in more detail, and especially knowing what collusion looks like in models, would give control protocols a detection and intervention point for monitor collusion that does not require any communication channel.

[1] https://academic.oup.com/antitrust/article/9/1/152/5880803 [2] https://arxiv.org/abs/2412.12480 [3] https://arxiv.org/abs/2507.14805 [4] https://arxiv.org/abs/2312.06942

Collusion is a serious risk for AI control, and our project would shed some light on how to detect whether collusion is happening. This will increase the effectiveness of AI control protocols once collusion starts arising in real life.

Mentees will build collusion scenarios on their own, and apply white-box techniques out of the box using existing trained J-space etc. See proposal for more. This is a greenfield project, with no existing codebase or collaborators.

• Highly proficient with Python and git (or Github).
• Has written code to run transformers from Huggingface, with custom system prompts.
• Has run code on GPUs from the command line on a cluster (or a local machine).