Whitebox Proxy Faithfulness for Blackbox Frontier Models

AI misbehavior gets harder to catch as AI becomes more powerful. It would be great to use whitebox techniques to diagnose them, eg probing for deception, eval awareness, power seeking. But we’ll never get access to the internals of frontier models, however, if features representations are universal enough, you could prefill a trusted whitebox with the generations of an untrusted blackbox, diagnose the whitebox, and make claims about the internals of the blackbox.

The high-level variables are model size, model family, and representations. We define “faithfulness” as how well the whitebox proxy can make predictions about the blackbox’s internal representations. A priori, we expect faithfulness to be “jagged”, e.g. deception vs sycophancy probing might have different faithfulness levels. Different techniques (eg NLA, AO, SAE, probing) will likely also be jagged and may require different validation methods. Downstream tasks will include probing for misbehavior and looking for nonverbalized reasoning in the cot.

After establishing proxy faithfulness baselines, we’d be interested in improving faithfulness. Distilling from the untrusted blackbox to the trusted whitebox is the most expensive, simple upper bound. There may be ways to make this more efficient, or narrowly improve faithfulness on important tasks. There are other ideas, like changing the whitebox’s representations in-context.

Stretch goal: One obvious failure mode would be the whitebox model failing to catch unverbalized reasoning in the blackbox model even given CoT, and we know the major frontier models also encrypt and summarize the reasoning traces. Measuring and mitigating the proxy faithfulness without reasoning, or with unfaithful CoT, is an important aspect we want to touch.m

This work is empiric and subject to change. The latest iteration of the project proposal is here: https://docs.google.com/document/d/1iKF2YSIANUviPkbHM2b3lPM0tNiDKoC3yOrt-0rixeU/edit?tab=t.0

mentor availability: (John Yan): I can be available for direct coworking and co-researching for up to 5 hrs/week/mentee. I expect them to be generally intelligent fast learners, willing to work 15-20 hours/week. Strongly preferred in person in SF. (Yuqi Sun): I can work 3-5h/week with detailed guidance and coworking (e.g. debugging blockers or brainstorm for methodologies, and not to promote for the situation but if we are very urgently facing a paper deadline i will jump in to help accelerate the work). Bay area in person preferred.

This adds interpretability techniques to the available tools for scalable oversight. The success of this work will enable better third-party monitoring and empower people outside frontier labs to secure their AI.

A weak version of success would look like: deception, sycophancy, powerseeking probes for effective runtime monitoring. A strong version of success would look like: improved pre-deployment elicitation techniques and improved post-hoc analysis on black swan events. A positive externality of this is one could use weaker models to diagnose stronger models, or enable model developers (including in the labs) to distill their strongest model into a cheaper model to make more expensive oversight techniques viable.

We’ve already done work where we use SAEs on a 27B model to look at generations of a 235B model, discover features that blackbox baselines could not, and validate them.

Mentees will essentially work as research engineers, we have well scoped plan but also encourage mentees to participate in literature review and ideation on specific methodology, we expect high agency of running planned experiments and discuss progress in weekly or biweekly meetings, mentees will also participate and gain experience in paper writing and publishing

Track record of long-term (>4mo) technical projects (usually predicted by 3+ YoE as a SWE/MLE) Python preferred Preferred prior publications/experience at research institutions Preferably in the domain of model training/interpretability, including experience of working with model representational space and model distillation

Prefer in person in Bay Area