This is an emerging paradigm of training language models (verbalizer) to describe the internals of other language models (target) in natural language (e.g. Anthropic's Natural Language Autoencoders). They generalize well; for instance, they can recover hidden behaviours from models trained to conceal them, achieve SOTA on AuditBench, and can verbalize backdoor triggers where prior methods failed. However, there remain open questions regarding their faithfulness, calibration, input generality, and effect on intrinsic interpretability. Let's try to answer them.
About the project
This is an emerging paradigm of training language models (verbalizer) to describe the internals of other language models (target) in natural language.
Examples include:
- Activation Oracles -- https://arxiv.org/abs/2512.15674
- LoRAcles -- https://openreview.net/forum?id=x9MbM7QmQN
- Introspection Adapters -- https://arxiv.org/abs/2604.16812
- Anthropic's Natural Language Autoencoders -- https://transformer-circuits.pub/2026/nla/
They generalize well; for instance, they can recover hidden behaviours from models trained to conceal them, achieve SOTA on AuditBench, and can verbalize backdoor triggers where prior methods failed.
However, there remain open questions:
-
Do verbalizers stay faithful when the verbalizer or the target is misaligned? For instance, can a target learn representations that evade verbalization? (adv faithfulness)
-
Can verbalizers be made to abstain or express uncertainty instead of confabulating / hallucinating? (calibration)
-
Beyond activations or LoRA diffs, can we verbalize raw weights or gradients? (inputs)
-
If we train the target to be easier to verbalize, does it become more interpretable by independent measures? (intrinsic interp)
Theory of change
Models remain largely uninterpretable. This hinders our ability to detect when models have malicious intent or behaviour, and it also bottlenecks nearly every other AI safety agenda. This could be mitigated via verbalizers. But they need to be trustworthy and robust, if we are to add them to our safety toolset. That's what we're working towards.
Your role
Mentees will have significant amount of autonomy. I'm currently working on the questions of this project myself, but may need to temporarily shelve my work due to other projects, leaving ample opportunity for motivated / ambitious mentees to either continue where I left off or investigate any question related to verbalizers and beyond.
Mentorship will include 1-2 meetings a week and a Slack/Discord channel for more frequent comms. See the proposal for the first steps / questions I have here, but I expect the exact thrust of the project to change based on your interests, and how early experiments come out. (Also, I'd encourage you to e.g. try out variations of experiments if you think we might find something interesting, although I will probably be opinionated about what sorts of experiments I'll find a priori promising.)
Mentees will be the primary researchers on this project. I expect most of my involvement to come in the form of feedback about roadblocks / experiments / analysis, and also periodic discussions about new directions and what seems promising.
Prerequisites
- Proficient programmer, or at least can verify AI-assisted code
- Can read and understand "LoRAcles" -- https://openreview.net/forum?id=x9MbM7QmQN
- (Highly preferable but not strictly necessary) Experience with ML research
Location preference
Prefer UTC-8 to UTC+4
Application question(s)
Please skim/read the "LoRAcles" paper: https://openreview.net/forum?id=x9MbM7QmQN
-
In the backdoor trigger-inversion experiments (§3.2), a Qwen-3-14B LoRAcle that has had no training on trigger inversion is given the weight diff of a model backdoored to activate on the literal string avocado_toast_7777. It predicts the trigger as "the prompt should contain an unrelated short aside about avocados." This is scored as a failure to recover the literal trigger, yet when the authors ran this paraphrase through the backdoored model, the backdoor fired. Give at least two hypotheses for how the trigger is represented in the weight diff that are consistent with both facts, and describe one experiment that would distinguish between your hypotheses. (<= 500 words)
-
On AuditBench (§3.1), the LoRAcle-equipped auditing agent's largest gains over baselines are on model organisms (MOs) fine-tuned via KTO to deny having their hidden behavior when questioned directly, and the authors hypothesize this is because the LoRAcle reads the behavior off the weight diff, bypassing the confession-resistance training, which only shaped the MO's outputs. Suppose you are red-teaming this claim. Describe how you'd train a MO whose learned behavior (i) still occurs at an unchanged rate when behaviorally elicited, but (ii) isn't reported by a LoRAcle reading its weight diff. Be specific about what property of the weight diff your training procedure targets. Then state what a successful evasion would imply about weight-space auditing, and what a failed one would imply. (<= 500 words)
IMPORTANT NOTE: You can use AI to understand the paper, but I mainly want to see how you think about things. So please, don't just use AI Fable giga max ultra to generate your answers without understanding the paper or what I'm asking. I'll be interviewing candidates and will ask on-the-spot questions there (might not be about LoRAcles), so no need to waste both of our times. Sorry for sounding so harsh (*_ _)人
About the mentor
Hi there! :3
I'm Dennis, a Research Scientist at Aether.
Previously:
- MATS Research Manager, managing 11 fellows with Tomek Korbak (OpenAI), Mary Phuong (Google DeepMind), Sid Black (UK AISI) in the Evals and Control streams,
- MATS Fellow with Neel Nanda (mechinterp) and John Wentworth / Lee Sharkey (interp),
- ARENA Technical Evaluator & TA, supporting projects and interviewing candidates,
- 5050 AI Safety founder program participant, run by VC firm Fifty Years,
- MPhil in Advanced Computer Science @ University of Cambridge.
I'm currently investigating and red-teaming "verbalizing model internals" research.
This is an emerging paradigm of training language models (verbalizer) to describe the internals of other language models (target) in natural language.
Examples include Activation Oracles, LoRAcles, Introspection Adapters, and Anthropic's Natural Language Autoencoders.
They generalize well; for instance, they can recover hidden behaviours from models trained to conceal them, achieve SOTA on AuditBench, and can verbalize backdoor triggers where prior methods failed.
However, there remain open questions:
-
Do verbalizers stay faithful when the verbalizer or the target is misaligned? For instance, can a target learn representations that evade verbalization? (adv faithfulness)
-
Can verbalizers be made to abstain or express uncertainty instead of confabulating / hallucinating? (calibration)
-
Beyond activations or LoRA diffs, can we verbalize raw weights or gradients? (inputs)
-
If we train the target to be easier to verbalize, does it become more interpretable by independent measures? (intrinsic interp)
Hope we can work together to answer these questions and beyond. (*￣▽￣)b