We train models to be more faithful by training their verbalizations to be consistent with their internal activations, e.g. "thoughts"
About the project
Modern day LLMs often demonstrate unfaithful chain-of-thought and unfaithful self-verbalizations. For example, when asked when they relied on a hint to answer a question, they would often answer “no”, when ablation experiments demonstrate that the answer is empirically “yes”. (https://arxiv.org/abs/2305.04388)
To mitigate this, we check whether we can train models to honestly verbalize their internal thoughts. This can be seen as a version of introspection (https://arxiv.org/abs/2511.08579) or confessions (https://arxiv.org/abs/2512.08093) training. This is distinct from the former work that simply trains models to interpret a decontextualized activation provided at the embedding layer, and distinct from the latter as we’re using model internal activations as ground-truth. The supervision comes cheaply from the internals themselves: probe readouts, feature activation values, or the measured effects of ablation and patching give us ground-truth labels for whether a verbal report is accurate.
Theory of change
Introspective reports are an easily-accessible interface for monitoring, and this proposal aims to improve the faithfulness of this interface.
Your role
See proposal for project details. I expect mentees to own their experiments end to end: implementing training and evaluation pipelines, running probing and ablation experiments, and analyzing results. I'll set the overall direction and help prioritize, but mentees will have real latitude in how they attack sub-questions, and I'd be excited for them to propose directions of their own. Mentees will likely start on a shared scoped starter task (e.g., testing how much introspection is achievable in-context) before splitting into complementary threads.
Prerequisites
- Highly proficient in Python and PyTorch.
- Have trained or fine-tuned a transformer language model (toy models or guided projects are fine).
- Comfortable reading empirical ML papers and reproducing methods from them.
- Familiarity with interpretability techniques (linear probes, SAEs, activation patching) is a plus but not required — willingness to learn them quickly is.
- Able to commit ~10 hours/week consistently through mid-December.
Application question(s)
- Please provide a critique of the following paper: https://arxiv.org/abs/2511.08579 (optional, but recommended; 400 words)
- Propose an initial experiment to test how much models can already verbalize their internal activations in-context, without training (300 words)
About the mentor
I am a researcher at Anthropic on the Interpretability Team, and an incoming Assistant Professor at the University of Chicago Data Science Institute. I recently completed my PhD at MIT CSAIL, where I was affiliated with the language & intelligence (LINGO) lab @ MIT, and before that I worked at Facebook AI Applied Research, and obtained my B.S. in Computer Science at the University of Washington.
My research focus is building AI systems with coherent, updateable, and interpretable models of internal and external phenomenon. Specifically, my research focuses on the following three types of models:
- World models: models of the external environment that update in the presence of new information and support coherent downstream prediction and reasoning.
- User models: models of the user's preferences, goals, beliefs, values, learning styles, and workflows.
- Self models: models of the AI system's own internal computations, external behaviors, and limitations. Together, these models enable AI systems to behave more reliably and predictably, in ways that are transparent and safe for humans.
Recently, I've been focused on training models to introspect and be self-consistent, although