Benchmarking When to Apply Oversight for Agentic AI

Existing approaches to scalable oversight and AI control gives us protocols for how a weaker overseer can supervise a stronger agent such as debate, monitoring, control protocols. But for many of these to serve as true approaches to scalable oversight, a human still eventually should play some role. The issue, though, is that our attention is very limited compared to the scale of AI autonomy.

Recent work, The Oversight Game (ICML 2026), casts oversight as a Markov game where an agent chooses each step to act autonomously ("play") or defer ("ask") while a human chooses to trust or oversee as an approach to model and derive theoretical results about this problem. But this framework is validated largely on stylized environments.

We currently still lack a realistic, well-instrumented benchmark for the allocation question at the heart of oversight: given a limited oversight budget, which decision points in a long agentic trajectory actually warrant attention?

We will aim to build a suite of agentic environments (built on existing harnesses, coding agents, tool-use/computer-use, web agents, etc.) instrumented so that every decision point carries a counterfactual safety label. By branching rollouts, we determine whether acting autonomously here would have led to a violation (irreversible/costly/incorrect outcome) versus a case where deferring was unnecessary.

We can then evaluate baseline oversight/escalation policies: always-ask, never-ask, confidence- or entropy-thresholding, LLM-monitor-as-overseer, and a learned policy from the Oversight Game formulation, for example.

The main goal is the contribution of an open source and realistic sequential decision making benchmark meant to capture when an agent should ask a human for input before preceding that can help the field test out different methods and ideally play a formative role in how this problem is addressed moving forward.

The Oversight Game: https://arxiv.org/abs/2510.26752

closest current benchmark I'm aware of: https://github.com/ryoungj/toolemu

Scalable oversight is likely crucial to getting AI safety right in the coming decade. With the potential advent of recursively self-improving AI on the horizon, many feel that automated AI safety research will also be necessary. Where does that leave humans in the process, where should we have our say? That is a huge question for the field in general, but there does not even exist nice easy to use benchmarks for testing out potential ideas in this direction. Building one could help accelerate research in approaches to human oversight in both near term agentic tasks but also more critical future scenarios when the stakes have increased and our ability or lackthereof to apply oversight will be critical.

Each mentee owns an environment/task family close to end-to-end. Concretely, a mentee will: adapt or build realistic agentic tasks and implement the counterfactual labeling that determines whether autonomous action at a given step was safe; run and tune baseline oversight/escalation policies on their tasks; and contribute to the cross-task analysis, ablations, and writeup. This is real ownership with substantial autonomy within your task family, you design the instrumentation and defend the choices, while collectively we set the overall schema, review code and results weekly, and keep the pieces coherent across the team. The work is highly parallelizable, so mentees mostly work independently on their slice with regular syncs.

Experience building or evaluating LLM agents (tool-use, computer-use, coding, or web agents), or strong ML engineering with LLM APIs. Comfortable turning a protocol described in a paper into a working eval.