Reliable Explanations of AI Behavior Across Functionally Equivalent Models

Mechanistic interpretability aims to explain how a model produces a behavior by identifying the neurons, features, or small collections of components involved. Researchers can then test an explanation by modifying those components, e.g. removing a neuron or replacing an internal signal, and checking whether the behavior changes.

However, neural networks can perform exactly the same computation in many different internal forms. Hidden components can be reordered, signals can be rescaled and compensated for elsewhere, or part of the model can use a different internal coordinate system. These changes, known as parameter symmetries, leave the model’s outputs unchanged but may change which individual neurons or directions appear important. An interpretability method might therefore seem to find different causes in two versions of what is functionally the same model.

We will study this problem using a small transformer with a controlled safety-relevant behavior, such as a backdoor that activates only when a particular trigger is present. Because we deliberately construct or train this behavior, we will have some knowledge of the computation that produces it. We will then use exact parameter symmetries to create many functionally equivalent versions of the model and carefully verify that their outputs and behavior remain unchanged.

We will evaluate one or two causal interpretability methods, such as neuron removal, activation patching, or automated circuit discovery. We will ask whether each method:

• identifies components that genuinely control the behavior;
• gives consistent explanations across equivalent versions of the model after accounting for the known internal transformation; and
• finds mechanisms that can be transferred between model versions while preserving their causal effect.

We will then develop a more reliable evaluation or method. For example, instead of treating one particular neuron as the explanation, we might identify a collection or direction of internal signals that transforms predictably when the model is rewritten. The strongest test will be whether a mechanism found in one version can be mapped into another version and still activate, suppress, or otherwise control the same behavior.

We will initially limit the project to one model, one controlled behavior, one family of exact transformations, and one or two interpretability methods. If the initial benchmark succeeds, we may extend it to a public safety model, sparse feature methods, or interactions among symmetries in the data, model parameters, and learned representations.

Interpretability methods may eventually help researchers and auditors investigate whether advanced AI systems contain backdoors, hidden objectives, deceptive strategies, or other dangerous internal processes. Causal interpretability is especially promising because it tries to move beyond correlations. However, an explanation may be misleading if it depends on an arbitrary choice of internal coordinates. If two models behave exactly the same but an interpretability method claims that completely different mechanisms cause their behavior, an auditor may not know whether the method found a genuine computation or merely a convenient description of one implementation. This could create false confidence in safety analyses or make results difficult to reproduce across model versions.

This project will provide a controlled benchmark in which the behavior and the model transformations are known. It will help distinguish genuine causal recovery from simply matching particular neurons or coordinates. It will also test whether explanations discovered in one implementation remain causally useful in another.

In the longer term, better tests of causal interpretability could strengthen internal audits of increasingly capable AI systems, help researchers evaluate interventions intended to remove dangerous behavior, and make safety findings more reproducible across training runs and model updates. This would complement behavioral evaluations and monitoring rather than replace them.

Mentees will have substantial ownership of the project’s implementation, experimental design, and analysis. After an initial onboarding period, they will help select or construct a small model with a controlled safety-relevant behavior, implement exact transformations that preserve the model’s outputs, and build tests confirming that the transformed models remain functionally equivalent.

Mentees will then implement or reproduce one causal interpretability method and design a benchmark for evaluating whether its explanations remain meaningful across equivalent model versions. They will analyze failed cases, develop appropriate controls, and help determine what should count as recovering the "same" mechanism when individual neurons and coordinates can change.

Based on the initial results and their interests, mentees will lead one extension, such as developing a symmetry-aware evaluation metric, identifying a more stable causal object, matching mechanisms between model versions, or testing whether a mechanism can be transferred while retaining its causal effect.

I will provide the initial mathematical framework, help select a tractable model and intervention method, meet with the team weekly, and give detailed feedback on experimental design, mathematical reasoning, interpretation, and writing. Mentees will be expected to maintain the codebase, document results, identify possible confounders, and propose next steps with increasing independence.

If the project has two mentees, one may focus on constructing the model organism and causal-intervention pipeline, while the other focuses on the symmetry transformations and methods for comparing explanations. Both will work toward a shared benchmark and research result.

• Highly proficient using Python.
• Have completed at least one project involving transformer training, fine-tuning, activation analysis, interpretability, or implementation.
• Spent at least 50 hours working with transformers.
• Working knowledge of linear algebra.
• Background in group theory is not required, but willingness to learn enough basic definitions (https://arxiv.org/pdf/2506.13018 Section 2, up to definition 2.4) to formalize our research questions is a plus.