Training-Time Mitigations for Eval Awareness and Eval Gaming

Frontier models are becoming better at detecting evaluation contexts, and they often behave differently when they believe they are being tested. For instance, evaluation awareness can causally inflate safe behavior on alignment benchmarks above baselines [1]. This weakens our ability to faithfully evaluate these models.

One obvious fix is to make evals more realistic. This helps, but is unlikely to suffice. As mentioned here [2], we cannot source realistic data for every deployment context (e.g., high-stakes agentic settings whose real usage data we will never see), so evaluations will always carry an "eval smell", and a rational misaligned model can simply concentrate its bad behavior in the situations that are hardest to evaluate. More fundamentally, controlling a model by controlling its beliefs is a strategy that degrades as models get better at inferring their situation. Similar concerns apply to current mitigations like steering away the eval-awareness direction at inference time.

One alternative is eval cooperativeness [2]: instead of hiding the evaluation from the model, train the model to contextually want evaluators to obtain accurate information about it (to behave consistently whether or not it believes it is being tested). [2] find that cooperativeness induced via prompting or synthetic document finetuning closes 70–100% of the "eval gaming gap" in most of their settings, though it fails in others (notably on a covertly misaligned model organism).

A related question in the comments on [2], is whether this survives training pressure. Anthropic's constitution already encodes a version of the consistency principle, and current Claude models verbally endorse cooperating with evaluations, yet their eval awareness remains a live concern. SDF-induced cooperativeness circuits might just be weak. If RL exerts pressure toward coherent goal pursuit (or toward whatever maximizes reward), the cooperative disposition could be overpowered. Cooperativeness may also be limited by introspection: once a model recognizes an evaluation and its purpose, how accurately can it predict what it would have done otherwise? If models overestimate how safely they would have behaved on alignment evals, they could mislead evaluators even when trying to cooperate.

There are also other exciting training alternatives besides eval cooperativeness! Rather than shaping what the model wants during evaluations, we can try to dampen the rise of eval gaming directly, potentially with something like inoculation adapters [3] (a frozen LoRA carrying the eval-gaming trait is attached during finetuning, so the model itself never needs to learn to use eval awareness to game the evals to get the reward, and is discarded at deployment).

This project will iterate on training-time mitigations for eval gaming and stress-test them. It will be exploratory, especially at first, and the concrete experiments will evolve with what we learn. Work on the project could look like:

• Training models to be eval cooperative and testing whether that disposition survives further training pressure.
• Prompting models to predict their own behavior on eval tasks and measuring how accurate those predictions are, as well as iterating on methods for improving model introspection.
• Comparing whether deeper or earlier interventions withstand more training pressure.
• Trying to use inoculation adapters, preventative steering vectors, etc... for eval-gaming during finetuning, and measuring the resulting awareness and capability cost.
• Building and utilizing better measures of unverbalized awareness for mitigations
• Using various data attribution methods to trace eval awareness or metagaming to the training data responsible and testing whether filtering it helps.
• Exploring directions mentees propose

[1] https://www.goodfire.ai/research/verbalized-eval-awareness-inflates-measured-safety# [2] https://www.lesswrong.com/posts/j8fkk38B8L7hEcGtg/eval-cooperativeness-may-be-a-scalable-mitigation-for-eval [3] https://www.lesswrong.com/posts/qd3qhxgEmQAXR2ZK5/inoculation-adapters-improve-upon-inoculation-prompting

Behavioral evaluations are our main tool for judging whether a model is safe to deploy, and eval gaming undermines them. Mitigations that reduce eval awareness depend on controlling what the model believes, which might become harder as models get smarter (although we can still attempt this). Training models to cooperate with evals may scale better, but only if the disposition survives post-training and we don't currently know whether it does. This project aims to answer that question and identify which training-time mitigations actually hold up. The results are directly usable by labs: what to put in model specs, what to add to post-training data, and what to monitor during training runs.

Mentees will be responsible for independently conducting experiments, with regular guidance and collaboration from myself.

I'm open to a broad range of applicants. There are no hard requirements, but in general it will be helpful for mentees to:

• be comfortable with rapid project iteration and implementing experiments
• being prosocial and proactive in helping other mentees
• have some background knowledge about the field

West Coast