Studying Catastrophic AI Misuse: Harm Uplift Measurement and Red-Teaming for Dangerous Knowledge

See this doc: https://docs.google.com/document/d/1LLvS6sdI1P-tcUIsqnbmMedGYlcRNC8upfnABJB3pLg/edit?tab=t.0

Models grow in dual-use knowledge alongside capabilities, and cheap extraction methods like task decomposition put that knowledge within reach of low-resourced actors, including terrorist actors already documented using it (Juelich 2026), while being difficult to monitor and mitigate. This raises the probability of catastrophic misuse, and we’ve identified two problems with standard operating procedures - providers struggle to detect task-decomposition attacks with current per-query defenses, and the field measures refusal rather than whether elicited output actually works. We see AI misuse in dual-use domains as an under-explored yet extremely significant risk, and we intend for this stream to help advance safety towards preventing practical harm uplift from AI. As such, we support many directions within this domain, and are particularly interested in studying the following:

1. Better jailbreak eval techniques like functional success rate (over attack success rate only counting refusal bypass) can help inform a proper harm uplift measurement
2. Alternative, benign, and publicly releasable benchmarks for harm uplift, that don’t contain hazardous information but reflect a strong relationship towards measuring harm
3. The detection and mitigation methods for these threat vectors, enriching the provider toolkit as these attacks scale and can also apply to oversight mechanisms for increasingly capable models
4. Improving automated red-teaming pipelines to strengthen defenses

This work advances AI safety by (1) accurately measuring and forecasting the risk of catastrophic AI misuse, (2) reducing the risk of catastrophic AI misuse and (3) improving detection/oversight mechanisms for increasingly capable models (this is important; see July 2026 OpenAI rogue deployment incident).

Each mentee can own one direction (above) and drive it. You pick the specific approach within that direction, run the experiments, and produce the analysis and writeup. John and Max set the overall agenda, hand you the prior red-teamer and datasets as a starting point, meet weekly to engage with your results and unblock you, help scope toward something publishable, and get work done if necessary. The goal is to end up with something publishable which you will have authorship on.

Required: high proficiency in Python. Comfortable calling LLM APIs and running open-weight models locally. Familiar with core AI safety and red-teaming concepts (jailbreaks, refusal training, evals). Able to work and deliver independently, and have clear reasoning transparency! Strong plus for specific directions: domain literacy in chemistry, biology, or cybersecurity for the dataset directions, experience designing or running AI evaluations, prior red-teaming or capability-elicitation work.