Forecasting the Outcomes of Long-Horizon Agents from Their Internal Representations

This project asks whether an AI agent’s internal representations can predict the eventual success or failure of a long-horizon task before the rollout is complete. We will run the same automatically verifiable tasks multiple times, collect residual-stream activations at selected layers and points in each trajectory, and train simple linear probes to measure when terminal-outcome information becomes accessible.

A key part of the design is comparing successful and failed attempts at the same task, which helps distinguish genuine information about the evolving trajectory from signals that merely encode task difficulty. We will also compare the probes against observable baselines such as elapsed turns, tool errors, and transcript features. The project builds on recent work showing that internal activations can support early failure prediction in shorter agent episodes [1] and that failure-related activation directions can be used to steer coding agents [2].

The immediate goal is to establish a tractable experimental pipeline and determine how outcome predictability changes across model depth and rollout time; longer-term, such signals could support more efficient agent evaluations and monitoring systems that interrupt, restart, or redirect deteriorating trajectories.

Time permitting, stretch goals may include training fine-grained linear probes on specific classes of failure and steering model activations to improve evaluation performance.

[1] Ruan et al., 2026. https://arxiv.org/abs/2607.06503 [2] Sui et al., 2026. https://arxiv.org/abs/2605.05980

Evaluations are essential for understanding the capabilities of new agents. Long-term tasks, especially in domains like cybersecurity and general coding, are increasingly important. However, these tasks demand large token budgets that can be financially prohibitive for many organizations. This work stands to improve the efficiency of evaluations while also exploring a better understanding of rollout failures through an interpretability lens.

Mentees are expected to take ownership of the project, though substantial guidance and mentorship will be offered by me. Mentees should be able to take charge of reviewing literature, spinning up work environments, running experiments, and analyzing results. If mentees are not familiar with one or more of these steps that is okay, as it is important to me that this is a learning experience for everyone involved. I am invested in providing mentorship to strong candidates, so an interest in learning and having fun is a plus!

• Strong Python proficiency, including experience debugging nontrivial research code, working with APIs or agent frameworks, and analyzing experimental data.
• Hands-on experience with transformer language models in PyTorch, such as running inference, fine-tuning a model, extracting hidden states or activations, or modifying a model’s forward pass. Experience gained through personal projects or tutorials is acceptable if the applicant can explain the implementation.
• Familiarity with modern language-model and agent research, including a basic understanding of transformer internals, tool-using agents, evaluation benchmarks, and common experimental failure modes.
• Comfort with empirical machine-learning research, including designing baselines, interpreting noisy results, checking for confounds or data leakage, and using held-out evaluation.
• Ability to work independently with imperfect research infrastructure. The project will involve debugging long-running agent rollouts, managing compute, inspecting failed experiments, and adapting the experimental plan when a benchmark or model does not -behave as expected.
• Sufficient time to contribute consistently, ideally at least 10 hours per week for the duration of the program, including attendance at weekly meetings and regular written progress updates.

Prior experience with mechanistic interpretability, activation probing or steering, agent benchmarks, distributed inference, or long-running GPU experiments would be especially valuable, but is not required.

Available for at least 3 hours between 9am-5pm at GMT-4 (Eastern US)