Building a Robust Activation Monitor

An activation monitor is a classifier that takes a model's internal activations as input and detects a property of the model's state or of its output. In its most common form it is a linear probe trained on residual-stream activations [arXiv:1610.01644] to detect properties such as deception [arXiv:2502.03407], harmful intent [arXiv:2310.01405], or hallucination [arXiv:2304.13734]. Monitors are attractive for safety for two reasons. They do not depend on what the model chooses to show, since a model whose visible output is benign may still represent internally what it is about to do. They are also cheap, one dot product per token, so they can run on every token of every response at deployment scale, where an LLM judge reading the same transcripts would be prohibitive. Early work established that truthfulness is linearly decodable [arXiv:2304.13734; arXiv:2212.03827; arXiv:2310.06824]. Recent probes separate strategic deception almost perfectly on their evaluation sets [arXiv:2502.03407], and frontier labs have begun deploying probes in production [arXiv:2601.11516].

In the last year, several groups independently found that high accuracy on a clean benchmark does not imply robustness. Much of what a probe captures is the surface text reflected in the activation vectors. When the textual evidence of the monitored behavior is removed, probe accuracy drops substantially, and on model organisms that never verbalize the behavior it falls close to chance [arXiv:2509.21344]. Probes fail to transfer across datasets and attack families [arXiv:2509.03888]. On organisms whose beliefs verifiably differ from their statements, every activation-based detector drops sharply [arXiv:2606.12618]. Models can be fine-tuned to evade them, and the evasion transfers zero-shot to monitors the attacker never saw [arXiv:2412.09565; arXiv:2512.11949]. The signal is distributed over several dimensions rather than a single direction [arXiv:2605.27958]. Some fixes to specific weaknesses have been proposed, for example multi-dimensional probes and style-augmented training data [arXiv:2605.27958], persona-coordinate projections [arXiv:2605.09391], follow-up questions [arXiv:2606.12618], conflict signatures [arXiv:2606.17229], and keyed activation watermarking [arXiv:2603.23171].

I believe that these weaknesses are consequences of the training recipe, and I propose to change the recipe. A monitor today is a linear classifier trained on a few thousand examples from one model, one concept, and one narrow distribution, and it monitors a model trained on trillions of tokens. Language models themselves became robust through scale and diversity of training data, and monitors were never trained this way. The one recipe-level intervention tried so far supports this reading, since augmenting the style of the probe's training data alone recovered near-perfect detection at every model scale [arXiv:2605.27958].

The object of the project is a single activation monitor trained across concepts, across models, and on context-augmented data. The monitor is conditioned on the concept it is asked to detect, so one model covers the whole concept list and can be queried on concepts it never saw. To train it on the activations of different models, each model gets a projection layer that maps its activation space into the monitor's shared space, and the monitor is trained on top of the projections.

Recent literature supports each part of this design, and no published work combines them. Behavioral directions align across model families once projected into a shared coordinate space [arXiv:2605.09875]. Affine maps transfer probes and steering vectors between models [arXiv:2506.00653; arXiv:2506.06609], and steering-based safety interventions transfer across Llama, Qwen, and Gemma [arXiv:2503.04429]. SAE feature spaces are similar across LLMs up to simple transformations [arXiv:2410.06981], refusal interventions transfer across architectures through a shared concept basis [arXiv:2601.16034], and in vision models a single sparse concept space has been trained over several networks through per-model projections [arXiv:2502.03714]. To my knowledge, no published work trains one monitor on many models' activations, and the closest paper describes cross-family alignment as an open problem [arXiv:2506.00653].

We will start by collecting a dataset for each concept and augmenting all of them with longer and more realistic contexts, embedding the labeled examples into real-world chats. We will then explore the design axes: which layer to take activations from, whether one or several, the architecture and the training parameters, how to condition the monitor on the concept text, and how to build the projection from each model's activation space to the monitor's space. Finally, we will evaluate on held-out concepts and held-out models, where a new model is onboarded by training only its projection while the monitor stays frozen.

The ideal deliverable is a single activation monitor that is model agnostic, concept agnostic, and robust to context.

Activation monitors are one of the few oversight tools cheap enough to run on every token at deployment scale, and they are starting to be deployed in production. The recent literature shows they are not ready to be relied on: they read surface text more than internal state, they fail out of distribution, and they can be evaded by fine-tuning. A single monitor trained across concepts, models, and contexts tests whether these weaknesses come from the training recipe rather than from the idea, and if it generalizes, the result is a single monitor, more reliable than current probes and reusable across models, with a new model onboarded by training only a projection layer.

Each mentee owns one part of the project: dataset collection and context augmentation for a group of concepts, one of the design axes of the monitor (layer selection, architecture, concept conditioning, the projection from each model's activation space), or the evaluation on held-out concepts and models. We will build the training and evaluation pipeline together, and all the experiments will run on it. I expect mentees to take ownership of the smaller design decisions, their code, and their runs, to interact with me through results, to ask for guidance when they are stuck, and to decide the broader design choices together with me. I already have a literature review of the field, and I will ask mentees to start from it. I will be generally available by text message, and we will talk once or twice a week depending on the need.

• Proficiency in Python, preferred if python for ML/LLMs like PyTorch, HuggingFace
• Comfortable reading interpretability papers
• Comfortable using AI coding tools to iterate fast.

Ideal:

• Has done or gone through the material of ARENA
• Experience with activation probing or steering, vLLM, Weights & Biases
• Some exposure so ML research