Coherence-preserving steering via learned activation denoisers

Background. Luo et al. (2026, https://arxiv.org/abs/2602.06964) train diffusion models on LLM residual-stream activations, producing a generative model of the LLM's internal states. (For those unfamiliar with diffusion models: you train a denoiser — take an activation, add a little Gaussian noise, teach a model to denoise it, add a little more, teach it to denoise that, and so on.) Among other things, they find that applying this model to activations that arise from steering improves coherence — and note that this is despite the corruption the denoiser was trained on (Gaussian noise) being different from what steering does (adding a vector).

The idea is quite simple. There exists a variant of diffusion models — Cold Diffusion (Bansal et al., 2022, https://arxiv.org/abs/2208.09392) — where rather than Gaussian noise, corruption is a deterministic degradation. That is much closer to what steering is: steering isn't randomized, it's a vector that gets added. So it seems reasonable that if we use this as the noise instead and train a model that way, we're likely to get better steering performance when applying it.

One risk: done naively, the model may just learn to undo the steering effect, which we don't want. We'll need to be smart about this — one idea is to train it to only partially undo the corruption.

We'll compare against the Gaussian-noise meta-model as the natural baseline.

Steering is a safety-relevant technique. Models can be steered to be less evaluation-aware, which lets us test them more reliably. Anthropic did this during the alignment assessment of Claude Sonnet 4.5, steering down eval-awareness directions to see how behavior changed. Models can also be steered toward honesty (e.g. Zou et al., 2023, Representation Engineering). The main practical cost of steering is that stronger interventions degrade output coherence, so methods that push out the effect-vs-coherence frontier make an important safety tool more usable. That's what this project attempts.

Science of LLM activations. A meta-model of activations is itself a model of LLM internals: building better ones — including understanding which corruption processes they can and can't repair — may teach us about the structure of the underlying object.

Mentees will take real responsibility for, and to a large extent charge of, the project. They'll be doing the brunt of the hands-on work — implementing the training setup, running experiments, and analyzing results — while I suggest directions, help prioritize, and give feedback throughout. I don't expect to be regularly doing direct work on the project myself.

Concretely: we'll meet at least weekly, potentially more often when there are results to discuss. I aim to respond within 24 hours and to give detailed feedback on anything mentees produce. So the working mode is high autonomy with fast, engaged feedback.

Strong scientific fundamentals: linear algebra, critical thinking, ability to design and interpret experiments. Coding ability per se less important; fluency with agentic coding tools (Claude Code, Cursor, etc.) matters more than writing PyTorch from scratch. Prior experience with mech interp or activation steering helpful but not required.

I would prefer the mentees to be in a similar time zone to another, ideally one that is not more than ~8 hours away from CET