Spot the Difference: Model Diffing with the New Interpretability Toolkit

Model diffing — characterizing what changed between two models — is becoming a central interpretability primitive: it underpins auditing fine-tunes, verifying that safety-relevant features survive model updates, and detecting drift in agentic systems whose behavior-level evals miss silently held state. But the toolkit is in flux. Crosscoders learn shared feature dictionaries across models; our prior SPAR rounds showed structural priors improve their feature matching ~30% NDCG@64 and produced a documented recipe for stable training. Meanwhile, cheaper instruments have just arrived: Anthropic's Jacobian lens (July 2026) transports any decoder's activations into vocabulary space, giving a model-agnostic shared coordinate system; logit-diffing methods identify domain-specific changes directly from output distributions; natural-language autoencoders offer another new lens. No one has systematically compared what these methods each see — and miss — on the same model pairs.

This project is deliberately open in structure: mentees choose a method (or comparison of methods) and an application, and own that slice. Example directions, not an exhaustive list: (1) benchmark learned diffing (crosscoders) against cheap shared-coordinate baselines (J-lens vocabulary space) on the same base-vs-fine-tuned pairs — when is the expensive method worth it? (2) apply diffing to detect what a targeted fine-tune changed, including changes invisible to behavioral evaluation (3) extend diffing to agentic settings — does agent training or long-horizon deployment shift internal representations in ways transcripts don't show? A returning senior mentee continues the crosscoder workstream; new mentees are not expected to build on it unless they choose to. Working pipelines, compute, and a stable-training recipe are inherited.

Target: workshop/main conferenc paper or preprint; methods and comparisons released open-source demo/webs.

Every safety-relevant question of the form "did this training change what we think it changed — and nothing else?" reduces to model diffing. As fine-tuning, distillation, and agentic post-training proliferate, cheap and reliable diffing becomes audit infrastructure: it lets providers and researchers verify updates, catch unintended trait transmission, and monitor agents for internal drift that behavior-only evals miss. Honestly comparing new instruments against learned methods tells the field which tools to trust for which job.

Each mentee owns one workstream end-to-end (experiments, figures, corresponding paper section) with a brief written weekly update. I set direction, review results in detail, and handle integration and framing. Primary workstream ≈80% of time, optional secondary support role ≤20%. High autonomy within a workstream, and scope changes agreed in meetings.

• Highly proficient in Python and PyTorch.
• Comfortable with HuggingFace Transformers and running GPU jobs independently (Colab/RunPod).
• Curiosity about evaluating new methods critically matters more than prior interpretability experience — SAE/crosscoder/TransformerLens familiarity is a plus, not required.
• Comfort with linear algebra (subspaces, norms, rank) and geometry (mainfold, projection) knowledge needed for the workstream.

No geographic restriction; mentees must be able to attend one of two weekly meeting slots anchored to Central European time (historically Saturday ~13:00 UTC and Wednesday ~14:00 UTC)