Mentees will develop new agentic biosecurity evaluations and extend existing ones toward longer, end-to-end tasks.
About the project
As models become capable agents, single-step benchmarks saturate and stop tracking the multi-step workflows that matter for biological misuse. This project develops new agentic biosecurity evaluations and extends existing ones toward longer, end-to-end tasks and paired designs to tease out the difference between a model's capability and its willingness to help.
Theory of change
Static single-turn benchmarks underpin most public claims about biological risk from frontier models, and they are saturating. If evaluation capability does not keep pace with agentic capability, the field loses its ability to detect the point at which models become useful across a full misuse workflow, which is the threshold most frontier safety frameworks are written around. Paired designs add a second thing developers currently lack: a capability gap and a willingness gap call for entirely different mitigations, and today's evaluations collapse them into one number. SecureBio's agentic evaluations are used in pre-release testing, so improvements here reach model developers quickly.
Your role
The mentee will own a task or task family end to end, from design through implementation to analysis, with the project lead setting scope and reviewing. Code contributes to a shared repository under our engineering norms.
Prerequisites
Strong Python, ideally with production or open-source experience rather than only research scripts. Has built something with an agent framework and debugged agent behavior. Inspect AI experience is a plus. Comfortable with Docker and running jobs on cloud compute. Willing to read biology protocols carefully. A biology background is welcome but not required.
Application question(s)
Please answer one of the below, 300-500 words.
- How should a managed-access program for bio-capable models be set up? What are the relevant parameters, what do you suggest, and why?
- Suppose a wet-lab uplift study is run. What kind of data would you want to collect and how would you propose those data inform subsequent in silico model evaluations?
About the mentor
SecureBio is a nonprofit biosecurity research organization specializing in technical research to mitigate risks from catastrophic pandemics. Our AI team develops rigorous benchmarks and evaluation frameworks to assess AI systems' biological capabilities, as well as mitigation strategies that can reduce risks once AI capabilities cross specific risk thresholds. We perform pre-release safety testing of frontier models (e.g. GPT-5.6), and our evaluations have been featured in the model cards of OpenAI, Anthropic, and Google DeepMind. Our work has also informed national security briefings and emerging governance standards.