Sandbagging (a model strategically underperforming on evaluations) is a growing problem not only for auditors but for downstream deployers and end users, almost none of whom have white-box access. This project builds and rigorously evaluates black-box, query-only methods for detecting strategic underperformance, centered on an adaptive auditing agent that probes capability across contexts.
About the project
Sandbagging (a model strategically underperforming when it detects it is being evaluated or observed) threatens any benchmark-driven safety or deployment decision, as well as regular model usage, because a strategically weak model invalidates the very scores that gate deployment.
We think this is a timely problem, for two reasons. First, it is not only an auditor's problem: third-party evaluators, downstream deployers building on an API, and end users all have a stake in knowing whether a model is actually trying, and mostly none of them will have white-box access. As an example we can think of claude fable first release, that was silently starting sandbagging and users had no information about it, however still relying on the outputs.
Second, we expect more of this in the future: as models get more capable and more evaluation-aware, strategic underperformance becomes more plausible, and the gap between how a model behaves when watched and how it behaves in deployment is exactly what black-box detection has to close.
Yet the recent literature shows the black-box toolkit is in poor shape:
Embedding-based lie detectors collapse under same-model controls, because they learn a model fingerprint, not deception (Yulong et al., ICLR ICBINB 2026). Trusted-monitoring judges correlate with answer incorrectness, making them competence classifiers rather than deception detectors. UK AISI's auditing game found black-box detection defeated by a model imitating a weaker one, with quirky-but-benign models triggering false positives; (Taylor et al., 2512.07810).
This project treats black-box sandbagging detection as an open problem and takes inspiration from red-teaming literature and persona-based jailbreaks to build adaptive, agentic settings to uncover red-teaming.
Theory of change
Reliable capability evaluations are key for every frontier-safety regime: dangerous-capability evals, deployment gates, they all assume the model is trying its best. Sandbagging silently breaks that assumption, and the dangerous-capability setting is precisely where auditors are restricted to black-box access. A black-box detector would directly strengthen the elicitation assurances that safety cases rest on. All outputs are intended for release.
Your role
At first mentees will be establishing experimental framework, replicating experimental setting. Next each mentee will own one workstream of developing black-box detector. Mentees make their own scientific judgments; strong contributors can co-author the resulting paper.
Prerequisites
High proficiency in Python. Have worked with LLM APIs or open-weight models (prompting, sampling, batched inference). Comfortable with classification metrics and basic statistics. Prior exposure to LLM evaluation, red-teaming/jailbreaking, or AI-safety auditing is a plus but not required.
Application question(s)
A claimed black-box sandbagging detector reaches 89% recall. What single control experiment would most cleanly test whether it is detecting deception rather than an artifact such as which model produced the text? State what result would falsify the deception claim. (200 words)
Propose an initial experiment, with a $1,000 compute budget, to test whether a query-only method can distinguish a sandbagging model from a genuinely weak model on a released benchmark. Specify the organism/models, what you would query, and the metric (and how you would set the false-positive rate). (300 words)
A capable model trained to imitate a weaker model's outputs defeats black-box detection. Explain why, and propose one probing strategy that might still work against it. (150 words)
About the mentor
I'm a Fellowship PhD student at EPFL DLab under Prof. Robert West, working on AI safety with a focus on the pretraining phase and interpretability. My research centers on personas inside LLMs, how behavioral dispositions form during pretraining, persist through post-training, and connect to safety, and I'm increasingly interested in how these ideas apply to trustworthy evaluation, including detecting when models strategically underperform. I am also collaborating with Apertus models to make safety pretraining at large scale. Before EPFL I was a research lead on uncertainty estimation and LLM factuality.
I currently supervise several MSc students at EPFL, including projects on persona vectors and jailbreaking. My style is largely high-level: we discuss ideas, how to better pose hypothesis and how to interpret results. Mentees own their workstream and make their own scientific calls, while I stay close enough to stress-test designs and help prioritize.