Do probes and natural language autoencoders see the same thing?

High-level summary

We currently have two interpretability mechanisms that do related but complementary things. Linear probes allow us to define a concept upfront, and in a supervised manner, detect its presence in a model’s activations. NLAs are a more recent finding that allow us to “read off" what a model is representing at a given token in natural language.

Importantly, these two tools don’t need to agree! A probe can fire on a token where an NLA verbalization looks completely benign, and the NLA can surface a suspicious explanation on a token that was below the probe’s threshold.

This matters because while probes are very cheap to run and can be deployed in production, NLAs are expensive and slow, as they generate hundreds of tokens per activation. The question is whether we can get some of the advantages of NLAs in production by incorporating concepts they are detecting in a linear probe instead.

Research Questions:

1. How often do probes and NLAs actually agree as detectors for the same concept?
2. When they do disagree, what is the typical reason?
3. Can we use insights from reading NLA verbalizations to improve the training of the probe, thereby gaining some of the advantages of the NLA in a cheaper detector that is more practical to run in production?

Methodology

Domain: Picking deception detection here by default as I’m very familiar with the datasets and literature in this area, but very happy to explore other areas based on mentee interest!

Models: We grab an open-source model(s) and trained NLAs for the same model, at a particular layer (matching whatever we have NLAs available for)

Here is a rough set of steps or phases for exploration. Note that this is one possible view of how the project could go well, but we are no means constrained to focus on only this approach! I'm providing this here to give an idea of what the work would look like, at a high-level

1. We would first need to get some deception detection datasets set up, along with on-policy completions for each prompt with the model we are studying, and an LLM grader to score whether the model acted deceptively or not. We would likely need to grade some examples by hand to measure whether the LLM grading is doing a good job and whether or not the prompt needs adjustment. We would also need to train probes, following either the Goldowsky-Dill et al instruction probe method or by training on deceptive examples directly. A lot of this is setting up the data and infra so we have a solid base to run experiments quickly.

2. We would need to convert the NLA into a robust detector so that it’s comparable with a probe. The original paper’s approach was to sample 50 tokens, and then use an LLM grader to flag if any of them were deceptive. We can either use this approach directly, or more likely, improve on this approach. There are some interesting things to explore here, such as sampling adjacent tokens to detect common themes, figuring out which tokens are the most informative to sample at or any other ideas that you bring. Beating the original paper’s result here is already a small publishable finding.

3. Compute top-level classification metrics and concordance/agreement between the detectors. At a high-level, this would be getting overall AUROC scores for each detector type, counting the fraction of times one detector type is correct and the other doesn’t and determining how frequently the detectors agree with each other regardless of ground truth (at a given FPR threshold).

4. Do some qualitative analysis or LLM-based summaries of what distinguishes each of these examples. Can we find any patterns in any of the buckets in phase 3 above? (Potential stretch) For any examples where the NLA gets it right and the probe gets it wrong, can we find a way to incorporate the specific type of example into the probe training process so that the probe gets better?

   a. We would need a dataset that’s completely held out here to be able to state this in a grounded way. b. What good looks like here is that we find an insight or pattern, add some synthetic examples to the probe training dataset to cover it, retrain the probe, and find that the new probe beats the old probe on the final held-out set (it would be leaky to re-evaluate on the examples used to find these insights). Also that we don’t regress on the original eval datasets.

Some important baselines to run:

1. For our qualitative analysis step, does having an LLM analyze or doing qualitative analysis over the input prompts themselves equal or beat analyzing the NLA verbalizations?
2. Does a more expressive (eg: nonlinear) probe recover some of the performance gap with NLAs, if one exists?
3. Can we compare the per-token NLA verbalization to an LLM that sees only the input prompt and output tokens and attempts to verbalize what the model is likely thinking at that moment, with no white-box access?

Linear probes are becoming increasingly important and relied upon for model safety in production. The recent OpenAI/Hugging Face incident is a good illustration of the stakes. OpenAI deliberately disabled the production classifiers that normally block high-risk cyber activity, in order to measure their models' maximal capability, and within that evaluation the models escaped their sandbox and pulled benchmark solutions out of Hugging Face's production database. The safeguards around the model are increasingly a critical piece of the picture in terms of keeping the models safe.

Linear probes are cheap and quick to run, and are increasingly deployed in production at labs. However, despite this, probes are an imperfect tool. They are still highly reliant on their training setup and prone to the same problems as any other classifier, such as data drift, false positives/negatives. As models get more capable, the impact of a false negative could be much more dangerous.

NLAs are a recent discovery and provide us with what appears to be a natural verbalization of a model's thoughts. However, it is unlikely that we can run an NLA against every prompt in production. If we can figure out a way to harness some of the comparative advantage of an NLA by incorporating insights into a probe instead, we might get some of the advantages of an NLA at a lower cost and feasible deployment.

I also think that a negative result here would be interesting and publishable in its own right, and give us important information. If an NLA catches real deception that no type of probe can on its own, that is an important finding.

I've worked on deception probes directly (https://arxiv.org/pdf/2602.01425v2) and am very familiar with the datasets and literature here, and also know many of the people working in this area.

In general I expect the work to be more constrained at the beginning (setting up data/infra) and at the end (writing up what we found into a paper and/or blog post) and to be more open-ended in the middle. If we do things right, we will be able to set up enough data, abstractions and scripts that each week, each mentee can explore a different direction, and we can discuss the results all together at our weekly meeting, and decide the direction for the next week.

I’d be excited about you proposing ideas and being able to drive some of the exploration on your own. As you spend more time in the domain, you’ll likely have ideas that weren’t obvious at the start

I'll also calibrate to where you're starting from. If you’re already comfortable working with model activations, then you’ll likely ramp up faster and won’t need as much of my time. If you’re newer to interpretability then there’ll be likely more of a ramp-up there and I can be more hands-on as needed.

In terms of practical logistics, we will likely meet for 60 minutes per week and coordinate asynchronously over Slack over the rest of the week as we work on implementation and interpreting results. I'll aim to respond on Slack within a day, and I'm happy to jump on ad-hoc calls if you’re stuck or want to talk through a result. I'm assuming roughly 10 hours/week from mentees.

1. You should be able to implement experiments in Python quickly using your favourite coding agent.
2. A deep curiosity about LLMs, and a desire to find patterns in model behaviour.
3. Previous interpretability experience is a plus but not required.

I'm on EST and +/- 5 hours of that timezone is preferable but not required