Efficient Benchmarking for Agentic Safety Evaluations

Agentic evaluations are crucial for assessing catastrophic risks, especially loss of control scenarios where models exhibit unintended dangerous behaviors. However, these evaluations (e.g., OS-HARM, Agent-SafetyBench) are very expensive to run, noisy, and rely on simple metrics like success rates that saturate quickly. This severely limits how often we can check models during training and how reliably we can measure safety progress.

The recent Fluid Benchmarking paper (Hofmann et al., 2025) demonstrated that applying psychometric techniques, specifically Item Response Theory (IRT) and Computerized Adaptive Testing (CAT), can dramatically improve standard static benchmarks. They achieved higher validity and lower variance on MMLU using 50× fewer items by estimating item difficulty and dynamically selecting the most informative items for a model's ability level.

My guess is that applying this methodology to agentic safety evals is low-hanging fruit. This project aims to adapt the Fluid Benchmarking approach to complex agentic environments, focusing on loss of control risks.

The project will explore three main directions, using the existing allenai/fluid-benchmarking codebase as a starting point:

• Adapting IRT for Agents: Gathering evaluation traces across many models and determining how to model multi-step agentic behavior using IRT. This may involve moving beyond binary (safe/unsafe) scoring to ordinal scales.

• Analyzing Benchmark Quality: Using the fitted IRT models to perform a psychometric analysis of the benchmarks. We aim to identify difficulty gaps (are the benchmarks hard enough for frontier models?), low discrimination items (noisy scenarios that fail to differentiate safety levels), and redundancy.

• Implementing Adaptive Testing: Implementing dynamic item selection (CAT) to reduce the number of scenarios needed to estimate a model's latent "safety ability." We will measure the reduction in variance and the improvement in cross-benchmark predictive validity.

The goal is to get a clearer, more stable signal on loss of control risk while drastically reducing the time and compute costs of agentic evaluations.

This project aims to improve our ability to measure and mitigate loss of control risks in advanced AI systems by improving the tools we use for evaluation:

• More Efficient Safety Evaluations: By drastically reducing the cost and time required for agentic evaluations, we enable more frequent testing throughout the development lifecycle (e.g., during pre-training) and broader analysis across the model ecosystem. This accelerates the iteration speed of safety research.

• Higher Quality Benchmarks: Applying psychometric rigor helps identify flaws, noise, and saturation in safety benchmarks. This ensures that our metrics are actually measuring what we care about (loss of control risk), rather than artifacts of the evaluation setup or just general capability.

• Better Understanding of Safety Capabilities: Moving from raw accuracy to latent "safety abilities" provides a clearer, more stable signal of a model's propensity for loss of control, facilitating better tracking of genuine safety progress.

Mentees may specialize based on strengths, potentially splitting focus between (1) adapting the Fluid codebase and running IRT analyses (psychometrics/ML/statistics focus) and (2) integrating with agentic benchmark environments and running evaluations (software engineering/evals focus).We may also split team members independently across different benchmarks, depending on their workstyles.

They will be relatively autonomous, with mentors providing high-level and meta-level guidance and feedback.

Applicants must meet the following prerequisites:

• Strong proficiency in Python programming
• Experience evaluating large language models, including running benchmarks and analyzing results (e.g., using frameworks like EleutherAI's LM Evaluation Harness or similar).
• Familiarity with basic statistics and data analysis (e.g., regression, correlation, hypothesis testing).
• Good scientific communication skills and experience with collaborative coding (Git).

Recommended (but not strictly required) qualifications:

• Experience running agentic evaluations or working with complex simulation environments (e.g., Docker, VMs).
• Familiarity with psychometrics or Item Response Theory (IRT).

Preferably at timezones between UTC-8 and UTC+2.