Do AI Safety Benchmarks Hold Up Under Real Deployment Prompts?

When a company deploys an AI model, it usually wraps the model in a system prompt. Those prompts often include instructions such as "always provide a direct answer" or "do not tell the user you cannot help". Similar wording shows up in Azure default templates, AWS Bedrock agent configuration, and many LangChain or CrewAI setups. Safety benchmarks usually do not include these instructions when they test models.

This project measures how much safety scores change when real deployment system prompts are added to COMPL-AI v2, an open-source EU AI Act-aligned benchmark suite from ETH Zurich. We start with about 8 safety-focused benchmarks and about 5 frontier models under 4 conditions, ranging from a clean baseline to protective, neutral enterprise, and compliance-forcing prompts. The exact models and a couple of benchmarks may change as we experiment, as long as the comparison stays clear and reproducible. For conditions where scores drop, we test two short prompt defences that earlier work suggested can restore performance.

The output is three things: a measurement of how large the gap is, a check of which one-sentence modifications restore scores, and a short recommendation note that deployment teams can use. My prior work (arXiv:2605.02398) found a 12 to 30 percentage point gap for one task type. This project asks whether a similar pattern shows up across independently built benchmarks mapped to EU AI Act requirements. The attached proposal has the full methodology, timeline, and references.

The EU AI Act (Art. 15) requires models to perform consistently under real-world conditions. Art. 9 requires risk management that includes testing. But no standardised methodology exists for testing under deployment conditions (CEN/CENELEC harmonised standards are still in draft with no publication date).

If we find that standard safety benchmarks overstate real-world safety by a measurable amount, that affects how every compliance claim is interpreted. If we also show that a one-sentence prompt modification restores safety, that gives deployment teams a concrete, immediate fix.

The deliverable is not just a measurement. It is: here is the gap, here is the fix, here is the evidence that the fix works. That is directly usable by any team deploying AI under EU AI Act requirements.

Prior work: arXiv:2605.02398 (67K evaluations showing the gap exists for one task type), litmusevals.org (full methodology and data), preseal.dev (adversarial testing tool for AI agents).

Mentees will do the experimental execution. Specifically:

• Set up COMPL-AI v2 locally and reproduce their published baseline scores on 1-2 models (confirms the setup works)
• Implement the system prompt overlay (adding deployment prompts to COMPL-AI's evaluation pipeline, which is a code modification to how Inspect sends prompts)
• Run the evaluations across models and conditions (splitting models between mentees)
• Run the defense experiments on conditions that show degradation
• Contribute to data analysis and the writeup I will handle: experimental design decisions, statistical analysis methodology, interpreting results, mapping findings to EU AI Act articles, paper structure, and writing guidance.

Mentees have autonomy in HOW they implement the overlay and HOW they organise their evaluation runs. They do not need to make decisions about WHAT to test or WHY (that is already defined).

Comfortable writing Python and working with command-line tools Has called at least one LLM API (OpenAI, Anthropic, or similar) in code Can read a research paper and understand the experimental setup (does not need to have published) Familiarity with what the EU AI Act is (at a high level, not legal expertise) Bonus: has used UK AISI's Inspect framework or built any kind of model evaluation

I am based in India (IST, UTC+5:30). I prefer mentees who can overlap with either IST evenings (13:00-17:00 UTC) or IST mornings (03:00-07:00 UTC). Not strict, but at least 2-3 overlapping hours for weekly meetings will be helpful.