Towards rigorous Alignment Evaluations.

Research Direction: Alignment evals try to measure propensities such as misalignment, scheming, sycophancy, and dishonesty. Their results feed system cards, safety cases, and deployment decisions. But propensities are much harder to measure well than capabilities. There is usually no ground truth (an LLM judge scores the behaviour), the constructs are vague (two published sycophancy evals rank the same models in opposite orders), and the behaviour itself is not a stable trait of the model: it shifts with context, framing, and the persona the model adopts, including whether it believes the situation is real. A science of evaluations is forming around problems like these (Apollo's agenda, Weidinger et al., UK AISI). This project asks its central questions about propensity evals: what would it take to trust these numbers, and how close do current alignment evals come? Starting Point: We begin with a deep audit of a single eval, most likely the emergent-misalignment measurement pipeline (Betley et al.): its protocol is reused across many follow-up papers, so findings propagate, and the fine-tuned models are open, so it is cheap to re-run. We build primarily on the two best-fitting existing instruments, the Agentic Benchmark Checklist and the ETH misalignment-evidence checklist, supplemented by construct-validity criteria from Bean et al.. The audit has a conceptual pass and an empirical pass. The conceptual pass grades the eval against the checklists: is the construct clearly defined, do the scenarios plausibly elicit it, do the claims match the evidence? The empirical pass reproduces the headline result in Inspect, then perturbs what the conceptual pass flags (judge choice, output format, scenario framing, score thresholds) to quantify how robust the reported numbers are. Early audits of individual slices (judge circularity, framing effects) suggest there is a lot to find. The pilot calibrates our checklist and tooling; from there we extend the audit to further suites such as Agentic Misalignment, MASK, and competing sycophancy evals. From the audits we aim to distill recurring flaws into practical guidance, a step towards a methodology for propensity evals that is grounded in documented failures. There is a genuinely open conceptual question here too: no existing instrument tells you how to quantify the validity properties that matter most for propensity evals, such as whether the model detects the eval, whether the judge is independent of what it grades, whether the measured trait is stable across framings, and how to power studies of rare behaviours. Fields that have long measured latent, fakeable traits like psychometrics are a useful reference point for this part. Natural extensions depending on interest include building an exemplar eval or testing whether eval behaviour predicts behaviour in deployment-like conditions. We're also very open to mentees who arrive with their own ideas in this space.

TOC: We view evaluations as the most important technical intervention towards a potential slowdown as well as to pass government policies and inform policy makers of AI threats. We also view evaluations as critical to determining whether our current alignment techniques are "working". However, neither of these effects can be realised if policy-makers nor lab leads "trust" our evaluations: Evals that understate misalignment provide false assurance for deploying dangerous systems. Evals that overstate it get debunked, eroding the credibility of alignment research with the policymakers and lab decision-makers it needs to reach. We therefore view this work as critical to securing that current alignment evals are taken seriously and measure what we want them to measure. Some work doing this with capabities evals at UKAISI is: https://arxiv.org/abs/2507.02825, who are interested in our project proposal.

At a high level, we expect mentees to be quite independent in setting up and running experiments, although we will be happy to help out with this towards the start of the project. Towards the first few weeks of the project we will have very concrete projects to start with, but would be very excited about mentees also thinking critically about areas of extension for this project with us, as although we currently have extension ideas we would be keen to mentor on other aspects of this project too.

We would like mentees who are able to run small experiments on LLMs in Python and think through problem critically. Skills that are helpful for this:

• Python
• Knowledge of transformers/LLMs
• Previous experience with LLM evaluations (not required)

We're happy to take on people who learn quickly, if they're ready to do a lot of technical learning independently! On the off chance that you have psychometrics experience this would also be extremely valuable (although certainly not needed).