Can we deploy real programs in Transformers with minimal cost to (or even gain to) inference and training efficiency?
About the project
Recent work has shown that a large fraction of attention heads in Transformer language models can be replaced with programmatic proxies with modest cost to perplexity and little effect on downstream task performance (https://arxiv.org/abs/2606.19317). That result is framed as post-hoc interpretability where the programs are explanations. This proposal bridges the gap from explainability to deployment: what would it take for programmatic QK circuits to run inside real, usable Transformer systems, at minimal cost to both task performance and efficiency?
This is potentially one step towards more ambitious interpretability, where we train and deploy natively interpretable models — in this case, a neurosymbolic Transformer. This can split into two separate workstreams for inference and training.
Theory of change
Interpretability today is mostly post-hoc: we train opaque models, then reverse-engineer them, and our tools chronically lag model capabilities. A safer trajectory is native interpretability — models built from components we can read. But existing approaches (e.g., weight-sparse transformers) buy interpretability by sacrificing capability, which means frontier developers won't adopt them. We try to tackle these challenges
Your role
See proposal. I expect mentees to own their experiments end to end. I'll set the overall direction and help prioritize, but mentees will have real latitude in how they attack sub-questions, and I'd be excited for them to propose directions of their own. Mentees will likely start on a shared scoped starter task before splitting into complementary threads.
Prerequisites
- Highly proficient in Python and PyTorch.
- Have trained or fine-tuned a Transformer language model (toy models and following guides is fine), and spent at least ~50 hours working with Transformers.
- Comfortable working with model internals: extracting and manipulating attention patterns, using forward hooks, or equivalent experience with libraries like TransformerLens or NNsight.
- For the inference workstream specifically: familiarity with GPU efficiency (Triton, CUDA, or FlashAttention-style kernels) is a strong plus but not required — one mentee with this background is ideal.
- Able to read and critique empirical ML papers independently.
Application question(s)
- Hard-fixing a head's attention pattern during training makes the QK path non-differentiable. Name two alternative ways to impose a programmatic pattern on a head during training, and give one failure mode of each. (200 words)
- Propose an initial experiment for either workstream. Be specific about the model, dataset, intervention, and what result would make you update. (300 words)
About the mentor
I am a researcher at Anthropic on the Interpretability Team, and an incoming Assistant Professor at the University of Chicago Data Science Institute. I recently completed my PhD at MIT CSAIL, where I was affiliated with the language & intelligence (LINGO) lab @ MIT, and before that I worked at Facebook AI Applied Research, and obtained my B.S. in Computer Science at the University of Washington.
My research focus is building AI systems with coherent, updateable, and interpretable models of internal and external phenomenon. Specifically, my research focuses on the following three types of models:
- World models: models of the external environment that update in the presence of new information and support coherent downstream prediction and reasoning.
- User models: models of the user's preferences, goals, beliefs, values, learning styles, and workflows.
- Self models: models of the AI system's own internal computations, external behaviors, and limitations. Together, these models enable AI systems to behave more reliably and predictably, in ways that are transparent and safe for humans.
Recently, I've been focused on training models to introspect and be self-consistent, although