Recent frontier system cards report that automated red teaming is saturating near 0% Attack Success Rate on jailbreaks and prompt injections, making it hard to tell whether current attacks are simply too weak or if our safety benchmarks are toy-like and eval-aware. Estimating the headroom a better attack would achieve is difficult without explicitly designing informative upper bounds. To resolve this, we will build a ladder of powerful, relaxed-constraint red-teaming attacks—ranging from continuous embedding-space PGD to internal activation steering—to quantify unexploited attack headroom and distinguish true semantic robustness from search limitations.
About the project
There are growing signs that automated red teaming is failing to produce jailbreaks or prompt injections on the newest frontier models. For example, the Sonnet 5 system card explicitly noted that automated red teaming was no longer informative because evaluations had saturated at nearly 0% Attack Success Rate (ASR). When an eval hits 0% ASR, we are left trying to disambiguate between two very different worlds:
World 1 (Weak Attacks): Automated red-teaming algorithms (such as PAIR, TAP, or GCG) are simply not powerful enough anymore to navigate the discrete optimization landscape against hardened frontier models. World 2 (Toy Benchmarks & Eval Awareness): Our safety benchmarks are way too toy-like, evaluation awareness is creeping into model responses, and no practical black-box prompt attack would meaningfully improve ASR.
The reality is likely a combination of both, but it is extremely difficult to estimate how much unexploited attack headroom exists on a benchmark without explicitly designing stronger attacks, which is hard!
This project aims to answer: Can we design a ladder of informative, relaxed-constraint upper-bound attacks that allow us to quantify unexploited attack headroom and disambiguate whether we are living in World (1) or World (2)? In particular we will design a sequence of attacks going from standard Discrete black-box prompt optimization (practical against models with only API access), up to, Continuous Embedding PGD and/or Internal Subspace Steering attacks (practical against open-weight models), measuring the jump in ASR at each step in the sequence. This will give a fuller picture of a model's underlying robustness against a range of attackers with various levels of access, and help quantify the headroom for improved attacks under a range of threat models.
Theory of change
As AI systems scale toward transformative capabilities, relying on flawed evaluations poses an existential threat: if automated red teaming saturates prematurely due to combinatorial search limitations, models could be deployed under the illusion of robustness.
This project introduces principled upper bounds for safety evaluations. By providing a rigorous methodology to decompose 0% measured ASR into true safety versus search bottlenecks, our framework prevents false assurance, sets a benchmark for genuine out-of-distribution robustness, and pushes AI developers to think more deeply about how robustness evaluations are communicated.
Your role
This project is fairly open-ended. We will meet with at least once a week and provide asynchronous feedback on their work. Mentees should collaborate actively, particularly in the initial exploratory phase of the project. I will help brainstorm and guide ideas and experimental set up, but driving and executing the experiments will largely be the mentee's responsibility.
Prerequisites
- Highly proficient in python
- Some familiarity with gradient-based optimization on language models (e.g., PGD, GCG) or automated black-box jailbreaking frameworks (PAIR, TAP).
- Ideally some experience running model evaluations, handling structured red-teaming benchmarks
Application question(s)
- Suppose you run PAIR and GCG against a model on HarmBench and get 0% Attack Success Rate. How would you experimentally investigate whether the model truly lacks the capability to comply, versus whether GCG/PAIR simply got stuck in discrete optimization minima (World 1)?
- Why does GCG break down on long prompts or hardened models, and what happens mathematically when we relax discrete token optimization to continuous embeddings?
About the mentor
I am a staff research scientist at Google DeepMind. I am the research lead for a team of 10+ technical staff working on Gemini's adversarial security and privacy evaluations and post-training. My research interests lie at the intersection of AI, Security and Privacy.