Investigating different types of scheming (sandbagging, alignment faking, etc) by creating model organisms that exhibit these behaviours, then testing modern interpretability techniques (NLA, J-space) to look for common structure. Such commonality would be a significant boost to the detection of naturally-occurring scheming.
About the project
We aim to investigate whether different types of scheming share any properties when viewed through interpretability lenses. Many different alignment failure modes can be described as strategic misrepresentation of the model’s own properties, conditioned on the model inferring that it is under observation (oversight). Sandbagging [1] is misrepresentation of capabilities, secret-keeping [2] is misrepresentation of knowledge, alignment faking [3] is misrepresentation of goals, and secret loyalties [4] are a misrepresentation of allegiance. Together, these behaviours fall under the umbrella term “scheming”.
In this project, we will train different model organisms that exhibit each of these alignment failure modes. We will consider creating model organisms through SFT, reinforcement learning, and perhaps using ideas from emergent misalignment [5], whose deceptive behaviors emerge through generalization and may be a closer analogue to naturally-arising scheming. Then, we will use white-box interpretability techniques to investigate whether there is any shared mechanism of concealment across these instances. We can also compare against black-box and standard white-box techniques (probes/SAEs). A common signal across different failure modes is likely not an artifact of any one model organism construction method; rather than detectors that latch onto how an organism was built and break in real-world cases, we hope this will lead to more generalizable scheming detection.
Our past experiments on sandbagging (one type of scheming) suggest that this behavior is encoded the same way when present on different types of content, but the direction does not transfer across mechanisms (e.g. SFT to RL). We hypothesize that shared structure may still exist beyond raw directions, at the level of nonlinear structure or natural-language descriptions of activations. We intend to explore interpretability methods suited to detecting these similarities, such as activation oracles [6], natural language autoencoders (NLAs) [7], and Jacobian-lens/J-space [8]. Finding such a signal would be a significant boost to detection of naturally-occurring scheming, for which we do not have enough real examples to train robust detectors directly. A negative result is also informative: it would supply additional evidence that detectors trained on model organisms should not be trusted out of distribution.
[1] https://arxiv.org/abs/2512.07810 [2] https://arxiv.org/abs/2510.01070v2 [3] https://arxiv.org/abs/2605.27681 [4] https://www.formationresearch.com/secret-loyalties-whitepaper.pdf [5] https://arxiv.org/abs/2502.17424 [6] https://arxiv.org/abs/2512.15674 [7] https://www.anthropic.com/research/natural-language-autoencoders [8] https://www.anthropic.com/research/global-workspace
Theory of change
As stated above, there is currently no general way to detect scheming that occurs across different types of model organisms. If new interpretability techniques grant this ability, it will make detecting scheming in the wild much more likely, reducing misalignment risks.
Your role
Mentees will implement different types of model organisms, and test with a few interp methods. The project is likely highly parallelizable because of these divisions (each can take one organism type, or one interp type).
Prerequisites
- Highly proficient with Python and git (or Github).
- Has written code to run transformers from Huggingface, with custom system prompts.
- Has run code on GPUs from the command line on a cluster (or a local machine).
Application question(s)
In your opinion, how long will it be until nearly all coding tasks can be automated by AI? Why do you say this timeline? Optional reading: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ (200 words)
What would be a simple and naive way to detect sandbagging using the new J-space interpretability technique? https://transformer-circuits.pub/2026/workspace/index.html (200 words)
In "Auditing Games for Sandbagging", what was the strongest type of sandbagging model organism? Do you think it would be possible to reverse its sandbagging? https://arxiv.org/abs/2512.07810 (200 words)
About the mentors
David lives in Canada (and often the UK) and enjoys AI safety research and mentorship. He works full-time as a Senior Research Manager at ERA, helping fellows execute research projects in each ERA AI fellowship. He also co-runs Lida Safety Research. Previously, David participated in the MARS program with Geodesic Research, conducted research at Mila, and was an early member of Yoshua Bengio's LawZero in Montreal. He was lead instructor at AI Security Bootcamp Singapore, and will serve the same role at the upcoming event FAST (Frontier AI Security Training) in Singapore.
David has previously mentored for SPAR, Athena, and other mentorship programs. He spent four years as a cyber insurance startup CTO, leading a 15+ person team. He holds a cybersecurity PhD from Columbia University and worked with 25+ students overall. David also works in AI risk communications, with a 30,000+ subscriber YouTube channel.
Linh is an independent AI safety researcher at Lida Safety Research, which she co-runs. Her research focuses on simulating AI governance policies to determine their effectiveness and feasibility. Previously, Linh worked on alignment at Mila through latent adversarial training for personalities. She participated in the MARS research program with Geodesic Research, working on chain of thought monitorability. Earlier, she did a postdoc at the University of Technology Sydney and obtained her PhD from the University of Queensland in natural language processing. Linh was a SPAR mentor in a previous iteration, and enjoys hackathons: she has won 1st place twice and 4th place once at Apart Research hackathons. She also won 4th place at the Redwood Research Alignment Faking hackathon.
I'm a PhD candidate in Mathematics at UCLA, interested in developing and understanding techniques in mechanistic interpretability, and applying them to a wide collection of problems in AI safety and alignment. I first came to this work as a SPAR mentee, on a project automating interpretability for per-layer transcoders through an agentic tooling framework, and I'm now co-mentoring with David and Linh on applying interpretability techniques to a range of model organisms.
Before moving into safety and interpretability research, I studied mathematics, statistics, and physics as an undergraduate and interned as a research scientist, a data scientist, and a quantitative researcher, working on topics such as agentic systems, causal inference, RL, and time series analysis. I'm currently finishing up my thesis at the intersection of differential topology and differential equations, and I'd love to see how those two areas can be applied to interpretability, safety, and alignment!