Emergent re-alignment: an error-correction approach to alignment.

Emergent misalignment (EM) is the phenomenon in which finetuning a model on a narrow set of bad examples — insecure code, bad medical advice — produces broadly misaligned behaviour on unrelated prompts (https://arxiv.org/abs/2502.17424). Existing mitigations are passive: mixing in aligned data (https://arxiv.org/abs/2506.19823), inoculation prompting (https://arxiv.org/abs/2510.05024), and data filtering all work by reducing the overall rate of misaligned generations. The model is never given the capacity to notice or repair misalignment once a response is underway — if a generation starts down a misaligned path, it stays there.

A stronger version of alignment is hence to teach the model to recognize misalignment in its own output, know what the aligned version would have been, and repair it. From a theoretical perspective, we can understand this as moving from passive error mitigation to active error correction. Namely, active error correction requires:

1. Syndrome extraction: recognizing from the partial output that generation has entered a misaligned mode

2. Correction: Updating its internal computation to to be consistent with an aligned response.

3. Readout: producing a final answer that reflects the correction rather than the error.

We will aim to implement a theoretical and practical version of all three stages.

Theoretically, we will extend a recently developed toy model of passive emergent misalignment mitigations (a small transformer trained on hidden-Markov sequences, in which a narrow misaligned data component shifts behaviour globally) to include the error-correction case. This requires appropriately modelling the effect of transitions between aligned and misaligned generation modes and looking for a regime in which a small number of transitions generalized appropriately. In the ideal case, we would be able to derive a phase diagram for active error correction to supplement the existing one for error mitigation.

Practically, we will attempt to embed this behaviour in a reasoning models chain of though. In initial experiments, naively finetuning on corrections in the chain of thought led to modest improvements in correction rate relative to regular alignment training. We will build on this by constructing active correction finetuning datasets for reasoning models. A simple approach is to optimize transcripts that transition from one to the other inside the chain of thought. We can benchmark the resulting passive and acive correction against against current passive baselines (aligned-data mixing, inoculation prompting) at equal data budget on standard EM evals, comparing both the rate of misaligned answers and the rate of successful mid-trace recoveries. We will measure the relative performance across all three stages of active error correction.

If active correction beats passive baselines at matched budget, and the toy model correctly predicts where it does this could give us a new new paradigm for safety training in which models actively seek to be aligned.

If active correction can beat passive mitigation, this would constitute a better defence against emergent misalignment — current techniques only lower how often EM happens, and a model that has started a misaligned answer has no way to recover. More importantly, it would inform how to do character training properly. Alignment training cannot anticipate every way a model might go wrong, so we ultimately want models that try to be aligned: models that know what good behaviour looks like, notice when they are drifting from it, and steer back (https://www.anthropic.com/research/claude-character). EM is the natural place to learn how, since it is the simplest known example of a persona change, driven by a handful of persona features (https://arxiv.org/abs/2506.19823). This would be a strong paper.

Mentees will work collaboratively in small sub-teams to derive theoretical results and design and implement experiments. I expect to spend time co-working together with mentees early on in the project (1-2 weeks), to craft a project plan, help run experiments, and work through the theory.

We will then collaboratively construct a more detailed project plan, and mentees will largely take ownership of the project, including conference submission. We will continue to meet weekly where I will make suggestions about next steps and co-work on bottlenecks when needed.

For the theory sub-team being able to work through the Hidden Markov formalism and be able to work through a 1L transformer optimizer update from a data set constructed this way.

Previous experience using agent for autoresearch is helpful, but not required.

For the application sub-team, being able to generate efficiently generate a dataset of CoT transcript will be necessary.

Anything can work, but I have a mild preference for the Bay area so we can co-work in person and I can provide snacks.