This project develops interpretability methods for latent reasoning models (e.g., Coconut) that reason in continuous embedding space rather than human-readable text, addressing a critical gap as this paradigm may become dominant over traditional chain-of-thought.
About the project
Traditional chain-of-thought reasoning in LLMs produces human-readable reasoning traces, enabling direct inspection, interpretability, and oversight. However, a new paradigm is emerging: latent reasoning, where models reason in continuous embedding space rather than discrete tokens.
Instead of decoding hidden states to tokens, the model feeds its last hidden state directly back as the next input embedding. This "continuous thought" can encode multiple alternative reasoning paths simultaneously. The model can perform breadth-first search implicitly, rather than committing to a single deterministic path.
The interpretability implications of this can be profound. However, latent reasoning may offer efficiency advantages that can drive adoption despite interpretability costs (yet to be proven at scale). We need to develop interpretability methods now, before this paradigm becomes dominant and we lose the ability to understand model reasoning.
This project aims to develop methods for understanding reasoning in continuous latent space. The core insight is that interpretability methods designed for sentence-level chain-of-thought analysis can be adapted to continuous thoughts. However, this adaptation is likely non-trivial and requires new techniques.
Research questions:
- Causal structure. Can we identify causal dependencies between continuous thoughts analogous to how we identify dependencies between chain-of-thought sentences? The causal graph framework should still apply, but the "nodes" are now continuous vectors.
- Anchor identification. Do latent reasoning models have "anchor" thoughts that disproportionately influence outputs? If so, how do we identify them without discrete boundaries?
- Decoding and probing: What information is encoded in continuous thoughts? Can we train probes to extract human-interpretable features from these vectors?
- Intervention methods. Can we develop principled interventions on continuous thoughts analogous to chain-of-thought interventions? What does "editing" a continuous thought mean?
- Faithfulness. Are continuous thoughts more or less faithful to actual model computation than discrete chain-of-thought? Does the lack of language bottleneck improve or hurt interpretability?
Theory of change
- Preserving interpretability as paradigms shift. Chain-of-thought interpretability has been a major win for AI safety. We can literally read model reasoning. Latent reasoning threatens to eliminate this transparency. Developing interpretability methods for latent reasoning before it becomes dominant can be crucial for maintaining our ability to oversee AI systems.
- Understanding faithfulness in new architectures. A key question for chain-of-thought is whether stated reasoning is "faithful" to actual model computation. Latent reasoning offers a natural experiment: without the language bottleneck, does the reasoning become more faithful (no need to verbalize) or less interpretable (no discrete steps to analyze)? Understanding this informs which paradigms to encourage.
- Enabling oversight of more capable systems. Latent reasoning may enable more capable intelligent systems (BFS search, better backtracking, fewer tokens). If such systems are deployed, we need oversight methods. This project develops foundational tools for that oversight.
- Informing governance and policy. The choice between human-readable chain-of-thought and efficient latent reasoning is partly a governance question. This research informs that choice: if we develop adequate interpretability for latent reasoning, the safety cost of adoption is lower. If we cannot, there's a stronger case for mandating readable reasoning in high-stakes applications.
Your role
Mentees will be primary researchers with substantial autonomy. They will build infrastructure (e.g., setting up Coconut training and analysis pipelines), develop interpretability methods for continuous thoughts, run systematic experiments to evaluate methods' effectiveness, and interpret results to draw broader conclusions. We expect active participation in weekly research discussions, proposing directions, and critiquing approaches. This project has a significant exploratory component, i.e., we don't know which methods will work best. Mentees should be comfortable with ambiguity and willing to try different approaches.
Prerequisites
Strong Python and PyTorch skills. Some familiarity with mechanistic interpretability and chain-of-thought reasoning will be preferred.
Application question(s)
Question 1: In Coconut, continuous thoughts can encode "multiple alternative next reasoning steps" simultaneously, enabling implicit breadth-first search. Propose a method to extract or visualize these multiple paths from a continuous thought vector. What assumptions does your method make, and how would you validate that your method is actually recovering the multiple paths rather than artifacts?
Question 2: One approach to interpreting continuous thoughts is to train a probe that predicts what discrete token the model would have generated if forced to decode at that step. Describe two potential failure modes of this approach for understanding what the model is "actually thinking." For each failure mode, suggest how you might detect whether it's occurring.
Question 3: Suppose you want to measure the "causal importance" of continuous thought c_2 for the final answer in a 5-step latent reasoning chain [c_1, c_2, c_3, c_4, c_5, answer]. In standard CoT, we might replace a sentence with an alternative. What is the analogous intervention for continuous thoughts, and what challenges arise that don't exist for discrete tokens?
Question 4 (Optional, link): Provide a link to code demonstrating your ability to train or fine-tune a neural network.
About the mentor
Uzay is currently a Research Fellow at Anthropic, working on mechanisms of introspection with Jack Lindsey and Emmanuel Ameisen. Previously, he was a research scholar at MATS, where he focused on the mechanistic understanding of LLM reasoning and chain-of-thought interpretability with Neel Nanda. Before transitioning to safety and interpretability research, Uzay was a technology entrepreneur with an exit, having founded ventures in healthcare and education. He was also a researcher at Mila, specializing in neuroimaging.