Preserving Chain-of-Thought Monitorability During LLM Post-Training

The current state of interpretability mostly involves using interpretability tools to understand model behavior, diagnose failures, and flag safety-relevant issues. However, we would like to move beyond post-hoc analysis and evaluate whether interpretability tools can also be incorporated into model-training loops to make advanced AI systems safer. Specifically for SPAR, we are interested in projects that use interpretability signals as reward models or regularizers, particularly in the context of RL post-training. We believe the questions below are concrete and sufficiently scoped for a focused SPAR research project, while addressing an important challenge for scalable oversight.

RL post-training is a useful testbed for this research because it is prone to reward hacking: models can learn behaviors that score highly under an imperfect reward function without satisfying the underlying intent. Chain-of-thought (CoT) monitoring is currently one of our most promising tools for detecting such behavior during training. Our long-term goal is to determine whether CoT-based signals can be safely used within RL post-training to discourage reward hacking while preserving our ability to understand why a model receives high reward.

The central safety problem is that optimizing against a particular CoT monitor invites Goodharting: a sufficiently capable model may learn to satisfy the monitor while concealing the reasoning or behavior that the monitor was intended to detect. This could train away one of the few channels through which humans can currently inspect a model’s reasoning. We are therefore interested in whether CoT monitorability can be made robust to optimization pressure. Our working hypothesis is that increasing CoT faithfulness—so that the CoT more closely reflects the computation that causally drove the model’s output—should improve detectability across diverse held-out monitors, rather than only against the monitor used during training. There are three broad questions we are interested in:

(1) What are the links between faithfulness and monitorability?

We adopt counterfactual simulatability as our working definition of faithfulness, following Counterfactual Simulation Training (CST; Hase & Potts 2026), and investigate whether improving this notion of faithfulness also improves safety-relevant monitorability. There are two specific sub-questions:

(a) Transfer: does faithfulness training generalize to post-training tasks, domains, reward-hacking strategies, and monitors beyond those used during training?

(b) Robustness under training pressure: when a faithfulness-trained model is subsequently placed under RL pressure that incentivizes reward hacking, does its CoT remain monitorable? In particular, can faithfulness training preserve evidence of reward hacking under conditions where direct training against a fixed monitor leads to Goodharting?

The primary outcome would be the detection rate of actual reward hacking by held-out monitors, rather than performance on the faithfulness objective alone. This distinction is important because a model could score well on a particular faithfulness metric while still learning to hide safety-relevant information.

(2) Links between model / LoRA size and CoT monitorability

One hypothesis is that smaller models, or models trained with lower-rank LoRA parameterizations, must rely more heavily on the CoT as working memory and may therefore be naturally more faithful and monitorable. In contrast, larger or more expressive models may be capable of optimizing the RL objective while keeping their visible CoT innocuous.

The trade-off is that smaller models or lower-capacity parameterizations may be too weak to solve the underlying task or fit the RL objective. The key question is therefore whether there exist model sizes or LoRA parameterizations that are expressive enough to perform the task, but not expressive enough to conceal reward hacking from CoT monitors. More broadly, this project would investigate whether restricting trainable capacity can preserve monitorability without substantially degrading useful capabilities.

(3) Efficient methods for CoT faithfulness regularization

The canonical method, CST, is expensive: it relies on LLM-generated counterfactuals and repeated simulator calls. We would like to investigate simpler alternative training procedures involving smaller judges, judge-free criteria, hidden-state-based signals, or distillation of CST’s faithfulness signal.

Building on CST’s use of LLM-rewritten faithful CoTs as positive training examples, we could run the full counterfactual-simulation and rewriting pipeline offline on a limited dataset, then distill the resulting labels, preferences, or rewritten traces into a lightweight faithfulness model. This lightweight model could then be used as a regularizer or reward signal during RL post-training.

The central safety question is whether the distilled signal preserves monitorability under optimization pressure, rather than becoming another proxy that the model can Goodhart. We would therefore evaluate distilled methods using the original expensive evaluator, diverse held-out monitors, unseen reward-hacking strategies, and direct behavioral measurements of whether reward hacking occurred. We would prefer a cheap and imperfect method that preserves hack detection under optimization to an expensive method that maximizes an abstract faithfulness score without improving safety.

Overall, our goal is to test whether CoT monitorability can be made an optimization-stable property of reasoning models. We would like to use CoT faithfulness as a training target while treating the detectability of reward hacking by held-out monitors as the ultimate safety-relevant objective. Positive results could provide a practical method for preserving an important layer of oversight during post-training; negative results could identify the interpretability conditions that train models to evade them.

Reasoning models often expose a natural-language scratchpad, creating a valuable opportunity to detect reward hacking, deceptive strategies, evaluation gaming, and other undesirable behavior during training. However, this channel is fragile. Once a chain-of-thought monitor is incorporated into a reward function, a sufficiently capable model may learn to produce monitor-approved reasoning while continuing the underlying undesirable behavior. In the worst case, post-training could actively remove the evidence that safety researchers rely on to detect misalignment.

Our theory of change is to investigate whether CoT monitorability can be changed from a passive diagnostic into a property that is deliberately preserved during optimization. We will test whether counterfactual faithfulness training causes the CoT to retain information that causally influenced the model’s behavior, whether this monitorability transfers to unseen tasks and held-out monitors, and whether it survives subsequent RL pressure that incentivizes reward hacking.

We will also study scalable versions of these methods, including small judges, probes, hidden-state signals, and distillation of Counterfactual Simulation Training. A low-cost signal that can be applied throughout post-training would be much more useful to AI developers than an expensive interpretability evaluation that can only be run occasionally.

Positive results could provide AI labs with a practical additional layer of oversight as models become more capable and are deployed in increasingly consequential settings. Negative results would also be decision-relevant by identifying when optimizing interpretability signals creates false confidence or teaches models to evade monitoring. We view this work as one component of a defense-in-depth approach to transformative-AI safety, complementing behavioral evaluations, red-teaming, control measures, and mechanistic auditing.

The mentee will have substantial autonomy and ownership over the project. With guidance from me and my co-mentor, Suraj Srinivas, the mentee will select and refine one of the proposed directions into a focused research question, formulate hypotheses, design the evaluation, implement the training and monitoring pipeline, run experiments, and lead the analysis and communication of the results.

We will help establish a tractable scope and a minimal decisive experiment at the beginning of the project. We will then provide regular technical guidance, feedback on experimental design, help diagnosing unsuccessful results, and support with research writing. However, the mentee will not simply execute a predetermined list of experiments: we want them to make meaningful intellectual decisions, challenge the project’s assumptions, and develop their own extensions as evidence emerges.

We expect the mentee to be the primary driver of the day-to-day research. Where the results are sufficiently substantial, we would work with the mentee toward a public report, workshop paper, conference submission, or open-source research artifact, with authorship reflecting each contributor’s work.

Applicants should have:

• Strong Python skills and practical familiarity with PyTorch.
• Hands-on experience with transformer language models—for example, fine-tuning a small model, training a probe, reproducing part of a paper, or building an evaluation pipeline. Following a guide is fine if you understand and can modify the code.
• Familiarity with basic ML concepts such as loss functions, train/validation splits, overfitting, distribution shift, and experimental controls.
• Some exposure to at least one of: LLM post-training, reward modeling, reinforcement learning, interpretability, or model evaluation.
• Comfort with Git, Linux/command-line tools, remote GPUs, and experiment logging.
• Ability to read an empirical ML paper and turn an open question into a concrete experiment.
• Reliable availability for SPAR and willingness to communicate blockers early.

Helpful but not required: Hugging Face/TRL, LoRA, activation probing, mechanistic interpretability, vLLM, or research writing. Prior publications, graduate study, and prior AI-safety work are not required.

pacific time zone, can range from 9am till night at 11pm