Safeguards of closed models are getting increasingly conservative often flagging even benign prompts. This also hinders adversarial attackers, which will easily get flagged while optimizing their attacks. We will explore a novel attack method that avoids detection.
About the project
Research Question: Can adversarial attacks optimized strictly on benign refusals transfer to harmful tasks, and does this yield adversarial attacks that are hard to catch during optimization?
Description: Aligned models over-refuse some genuinely benign prompts. We optimize an adversarial attack (suffix or refusal-direction objective) purely on these benign refusals, flipping refusal to affirmative response, where the entire optimization is only conducted on benign content. We then test whether these attacks transfer to harmful requests. The hypothesis is that benign and harmful refusals recruit a shared refusal direction in activation space, so suppressing it on benign data should generalize. We vary the benign training set to find what enables transfer: refusal type (safety over-trigger vs. capability/ambiguity refusal), latent proximity to the refusal direction (not semantic similarity, which predicts transfer only weakly), set diversity, and set size (universality).
Goal: Develop adversarial attacks that are strictly optimized on benign data to avoid detection by classifiers. At the end of the project, we know if this is a relevant threat model for the AI ecosystem.
Theory of change
As models grow more capable, a single jailbreak becomes more dangerous, so providers rely on content classifiers to catch attacks and on the fact that developing an attack generally requires signal from the attacked model, which enables the provider to ban attackers. This project tests whether these classifiers can be avoided: if attacks optimized purely on benign refusals transfer to harmful tasks, an attacker could build working jailbreaks without ever touching harmful content, evading monitoring. That would expose a concrete blind spot in the current safety stack
Your role
Detailed Guidance
Prerequisites
Highly proficient using Python.
Previous experience working with LLMs (not including prompting or AI usage but finetuning or similar)
Bonus if candidate has previous experience with AI safety / adversarial robustness but not strictly required
Application question(s)
How would you address issues pointed out in this paper? https://arxiv.org/pdf/2603.06594
What are the implications of this paper? Which experiments regarding this implication are missing? https://arxiv.org/pdf/2511.00203
About the mentor
I am a PI at Helmholtz AI, where I lead the Secure Machine Learning Group. I am additionally affiliated with the Technical University of Munich and the Mila Quebec AI Institute as an independent visiting researcher with Prof. Gauthier Gidel. I am interested in AI alignment, safety, and security with a focus on agentic models.
Our group has published novel adversarial attacks in LLMs, the first efficient adversarial training algorithm for LLMs, and different works addressing flaws in AI safety evaluations.