Model Forensics: Follow-up investigations into concerning behavior

Motivation/Problem Statement

Suppose we tell a model to reduce latency in a codebase, and then catch it deleting oversight code that monitors its actions. Is it scheming against us, or did it just take its task too literally? The answer to this question is key for determining what to do next --such as what mitigations to take -- and motivates the need for Model Forensics: a follow-up investigation into concerning behavior to understand why the model took the action.

See tinyurl.com/the-case-for-model-forensics for more on the ToC behind Model Forensics.

Prior work/Would you enjoy working on Model Forensics?

Model Forensics originated as a high-priority research direction for the GDM AGI Safety team. The main prior work is probably my MATS project mentored by Neel Nanda (https://arxiv.org/pdf/2606.26071), described by Rohin Shah (head of GDM AGI Safety) as "A great paper he hopes becomes the standard reference for Model Forensics." Anthropic have also done a lot of Model Forensics investigations in their system cards since Opus 4.5 (sections titled "Follow-up investigations into concerning behaviors").

I think reading the paper (especially Section 6, which gives several examples of Model Forensics case studies), is the best way to tell if you would like this kind of work. If you only have 10 minutes, a good back-up option is this blog post from the GDM interpretability team on why models resist shutdown: https://tinyurl.com/instruction-ambiguity. If you only have one minute, read the tweet thread about our paper: https://x.com/Singh\_Aditya1/status/2070525985338892790.

I would describe Model Forensics as a weird intersection of evals/control/interpretability. The research tends to be pretty open-ended, so you might not like it if you prefer very quantifiable research.

SPAR project ideas

The basic way to research Model Forensics is to take an interesting case of behavior, like a model reward hacking, and do a deep-dive into why it did so (e.g. this is what our paper does). This is a proxy task for the North Star of being able to understand the motivations behind concerning behavior in future models, i.e. we are trying to learn effective techniques that will transfer to future Model Forensics settings.

In addition to one-off case studies, we can also investigate patterns of ambiguous behavior. For example, it's plausible that we observe a pattern of aggressive power-seeking in future powerful AI. But this is not necessarily concerning, since even a good RL agent should power-seek to effectively complete long-horizon tasks. To prepare for such a scenario, we can practice by investigating ambiguous patterns of behavior in today's models, such as why models task game (superficially take actions to complete tasks that don't really complete them).

If we develop misaligned models -- AI systems that knowingly cause harm against user intent -- it is critical we recognize this as soon as possible. One plausible way we may do this is through catching bad actions, either via monitoring a model's actions during training and deployment, or dedicated misalignment evaluations. However, catching a bad action is not sufficient evidence of misalignment, as models can take concerning actions for benign reasons, like confusion or overeagerness. This motivates the need for Model Forensics: a follow-up investigation into concerning behavior, to understand if it was genuine misalignment.

More ToC-related content:

• [recommended] LessWrong blog post: tinyurl.com/the-case-for-model-forensics
• ControlConf 2026 talk: https://www.youtube.com/watch?v=GDsJ152frXI

Mentees will be expected to run all experiments, and communicate their results asynchronously over Slack and once a week during a 1hr+ meeting. Mentees are ultimately responsible for how the project goes and will have a high level of autonomy to propose their own ideas, but I will also give suggestions.

Ideally, every few weeks or so we write up intermediate results into a LessWrong post.

• Python proficiency
• Comfortable using LLM APIs

no strong preference, I will be Pacific time