AI character evaluations

Frontier labs are increasingly publishing model specs and constitutions as their stated alignment target. The details of this target, and the degree to which the model is aligned to it, could have huge consequences for AGI and ASI outcomes (https://www.forethought.org/research/the-importance-of-ai-character).

Often, the propensities detailed in a model spec are referred to as AI “character” traits. In this project we will build a measure of one specific character trait (likely in the ball-park of: behaviour in extreme concentration of power scenarios, corrigibility, prosocial tendencies, law-following, or tendency to escalate to humans/whistleblow). We will err on the side of being very specific with the behaviour we wish to measure.

The first stage of the project will select the trait to focus on. We will score each candidate behaviour based on the neglectedness, impact, and the tractability of producing a SOTA eval. After selecting the behaviour, we will create a 1-pager containing our literature review, theories of change for working on the eval, initial plan and key questions we want to answer with our eval. I will work closely with the mentees here.

To produce the eval, we will lean heavily on automated alignment evaluation tools such as Petri, and likely can start with this code: https://arxiv.org/pdf/2605.24229. The mentee should gain a hands-on understanding of the strengths and weaknesses of these existing methods. The mentee will then explore improving upon the off-the-shelf eval via the following levers:

• Improving the seed information and context provided to the auditor: This could include detailed descriptions of the behaviour we are trying to measure, and descriptions of effective methods for red-teaming the models behaviour.
• Designing custom high quality environments, using early automated auditing results to inform what environments the evals should be seeded with, figuring out how to add realistic details to environments where needed.
• Improving the grader to ensure it measures exactly what we want
• Customizing the automated auditing pipeline where needed

The evals should adequately explore the boundaries of the model's behaviour, distinguish different models, investigate the factors that influence the model decisions, and measure how models weigh these factors.

We will build a high quality measure of the character trait first. From there, we can consider mapping this measure into a “model spec adherence” metric.

The project should aim to produce a workshop paper. The paper should contribute both novel insights into how models behave with respect to the character trait, and insights into eval design and pipelines that can be extended to evals for other character traits.

Building good measures of a model’s character traits is important to: (1) Provide evals that help the labs spot flaws, or that the labs can hillclimb when aligning their model (2) Inform researchers, policy-makers, and the wider community about current AI behaviours and values (3) Hold labs accountable to instilling beneficial character traits into their models, and to adhering to their public model spec.

See proposal. The mentee will be expected to handle most engineering and low-level implementation themselves. I will work closely with the mentee in the initial project scoping phase, but after this will expect the mentee to have reasonable levels of autonomy between meetings and to drive the project where they can.

• Has basic context on AI safety (e.g., equivalent to having taken a BlueDot course)
• Has previously successfully executed on an empirical AI project (e.g., lead author on a workshop paper-level output)
• Has spent 80 hours doing technical AI safety research

Preference for US-friendly timezones