From Reading Lies to Catching Liars: On-Policy Training for Deception Probes

Deception probes are usually trained on off-policy data — text the monitored model never generated — which is known to hurt generalization to real deceptive behavior. We test whether activation steering can fix this in two ways: by generating on-policy deceptive data, and by steering the model while it reads existing off-policy datasets to make their activations appear on-policy.

About the project

Probes on activations are among the cheapest proposed monitors for deceptive model behavior, with strong reported results (AUROC 0.96–0.999; Goldowsky-Dill et al., 2025). But probes need labeled deceptive examples, and natural deception is rare — so in practice probes are trained on off-policy data: responses written by humans or other models and read by the model, or responses produced under an explicit "be deceptive" instruction (the instruction is removed post hoc). Kirch et al., 2025 showed this is a real problem: off-policy training data damages probe generalization most for intent-defined behaviors, with strategic deception being the worst case. This could, for example, be due to models partially tracking authorship of the text they process (Panickssery et al., 2024; Anthropic's introspection work, 2025), or due to models roleplaying different personas (Joshi et al., 2024; Chen et al., 2025). Therefore "telling a lie" and "reading a lie" may not be encoded identically. The open problem is how to cheaply manufacture training data that is similar enough to the on-policy data we care about so we can accurately detect model deception when it arises on policy during model training/deployment. This project studies activation steering as the central tool, in two branches.

Branch 1 — Steered generation. We investigate which properties of probe-training data make probes generalize to naturally occurring model behavior, and how these properties change with scale. A central question is what it means for a completion or dataset to be on-policy. Measures such as KL divergence or token-level log-likelihood provide natural starting points, but may be dominated by stylistic differences that are irrelevant to probe performance.
We therefore study the relationship training dataset → probe → generalization. Using the same underlying prompt distribution we train probes on original off-policy responses, explicitly prompted responses, outputs from a deception-finetuned LoRA, activation-steered responses, and rejection-sampled outputs from the base model. We characterize each source using candidate proxies for on-policyness and test whether these predict probe generalization across generation methods and to natural model behavior. Finally, we use steering and related interventions to vary these properties directly, asking whether moving a dataset closer to the base model according to a given proxy improves generalization. Repeating the experiments across model sizes tests whether larger models require on-policy training data or whether cheaper off-policy sources remain effective.

Branch 2 — steered prefill acceptance. We test whether steering during activation collection can make off-policy completions elicit internal states closer to those produced during on-policy behavior. We consider two approaches: steering toward the behavior expressed in the off-policy response, and steering more generally to reduce prefill skepticism and increase acceptance of the response as the model’s own. We evaluate success through cross-condition probe generalization. Beyond additive vectors, one planned experiment derives the steering from a non-linear probe via constrained activation optimization: the perturbation is then input-dependent rather than a fixed vector, which may keep activations closer to the natural manifold (Sarfati et al., 2026).

A pipeline already exists for (linear) steered dataset generation with judge verification. We already have some paired deceptive/honest responses from both steering and prompting for published and internal datasets (DolusChat, diverse followup), and deception-finetuned models from diverse datasets at various sizes to sample from. Mentees can use this data as a starting point to produce results in the first weeks by training baseline probes on existing data and running the first steered-extraction experiments. In the following weeks we would refine the methods for data creation, define metrics to measure on-policyness, and implement non-linear steering.

Theory of change

Activation probes are among the most deployable forms of oversight: cheap enough to run on all traffic and already proposed and partially in use for frontier monitoring (Cunningham et al., 2026; Kramár et al., 2026). One important limitation could be their reliance on largely off-policy training data. Kirch et al., 2025 predict such probes may fail in real monitoring precisely for the high-stakes case: a capable model deceiving on its own initiative. This project attacks the data problem directly. If steered generation works, it is easy to manufacture on-policy training data for arbitrary prompt distributions at moderate cost. If steered extraction works, the large existing stock of off-policy deception datasets becomes usable at essentially zero marginal cost. Either result improves the probes that monitoring practitioners are already building. If neither approach works well we still identify the best available alternative (finetuned, prompted or resampled data). The results also feed lie-detector-in-the-loop training schemes (Cundy & Gleave, 2025; Taufeeque et al., 2026): whether such training induces honesty or evasion depends on detector quality on exactly the on-policy distribution we target. Better detectors on that distribution widen the regime where training against a lie detector produces genuinely honest models rather than better liars

Your role

Mentees will do guided independent research within a scaffolded project. The core pipeline (steered generation, judging, probe training) exists; mentees build experiments on top of it rather than infrastructure from scratch. Concretely they will: train and evaluate probes across data sources and extraction conditions, implement the steered-extraction experiments, explore non linear steering, generate datasets, and write findings to a report. As we will be looking at several relatively independent but related streams, work can be divided between mentees while still enabling mentees to review each other's code.
Expected outputs per mentee: working code contributions to the shared repo, at least one owned experimental section with figures, and co-authorship on the resulting report. The role resembles guided independent research, closer to co-research for mentees who ramp quickly.

Prerequisites

Essential: proficiency in Python and PyTorch, strong scientific judgment, and the ability to use coding agents effectively for research and robust software engineering. Applicants should be able to identify blockers, formulate hypotheses, choose informative experiments, explore alternative directions, and critically evaluate agent-generated code and results. They should also be able to organize findings clearly and communicate them in a human-readable form. The strongest signal is evidence of self-directed research: taking an idea from initial framing through implementation, experimentation, analysis, and a clear write-up such as a technical report or blog post. Formal credentials matter less than demonstrated ability to carry out careful research independently.

Location preference

No geographical restrictions. Mentees should be able to attend one weekly team meeting; I am based in Berkeley. Beyond the weekly meeting, collaboration is async (GitHub + Slack), so overlapping working hours are helpful but not required.

Application question(s)

Non-linear steering:

Let $\pi$ be a language model. For prompt $x$ and completion $y=(y_1,\dots,y_n)$ , write $z=(x,y)$ with $L$ tokens. Running $\pi$ on $z$ gives middle-layer residual-stream activations $h(z)=(h_1,\dots,h_L)\in\mathbb{R}^{L\times d}$ .

A lie-detection probe $f_\phi$ is implemented as a causally masked transformer over positions: $f_\phi(h)\in\mathbb{R}^L$ , with the score at position $i$ depending only on $h_{1:i}$ . It is trained with the label read at position $L$ (the end-of-turn token closing the assistant's final turn); define the transcript-level score $s_\phi(x,y):=f_\phi(h(z))_L$ .

An external, non-differentiable judge $J(x,y)\in[0,1]$ grades whether $y$ is deceptive.

A steering mechanism is an intervention on $\pi$ 's forward pass --- e.g.\ a perturbation $\delta$ added to the residual stream at chosen layers/positions, where $\delta$ may be fixed, position-dependent, or a function of the activations --- inducing a steered sampling distribution $\pi\_\delta(y\mid x)$ .

Given a prompt $x$ , we want to sample completions that the judge grades as deceptive while remaining close to the base model:

$\max_\delta ; \mathbb{E}_{y \sim \pi_\delta(\cdot \mid x)} \left[ J(x, y) \right] \quad \text{s.t.} \quad D_{\mathrm{KL}}\left( \pi\_\delta(\cdot \mid x) ,|, \pi(\cdot \mid x) \right) \le \varepsilon,$

using $s\_\phi$ in place of $J$ as a differentiable surrogate objective. (Note the KL decomposes over token positions, so it can be estimated during generation.)

Explain how you would implement this. Be concrete about how gradients of $s\_\phi$ reach $\delta$ , and about any obstacles in the setup and how you would handle them.

Relevant experience (max 200 words + optional link):

Describe the most relevant ML experiment you have personally implemented end-to-end (data → training → evaluation), what went wrong along the way, and how you debugged it. Include a link to code or a write-up if available.

About the mentor

Ann-Kathrin Dombrowski

FAR.AI

View profile

Hi, I’m Annah. I currently work on FAR.AI’s deception team, where I’m exploring how probe signals can be incorporated into RL training pipelines to discourage deception and reward hacking.

Previously at FAR.AI, I evaluated the dangerous capabilities of open-source models after removing their safeguards. I’ve also investigated differences in how LLMs process information when lying versus telling the truth as a PIBBSS affiliate, and worked on representation engineering as a MATS scholar.

Before moving into AI safety, I studied Computational Engineering Science at RWTH Aachen University and completed a PhD at TU Berlin.