Internal monitoring when chain-of-thought becomes illegible

Motivation CoT monitoring only works if a monitor can read the reasoning. Korbak et al. (2025) argue this is a real but fragile opportunity for AI safety, and that shifts in training practice, such as scaled-up reinforcement learning, threaten to degrade it. That degradation has a name and a growing evidence base: illegible CoT, where reasoning drifts into compressed, encoded, or otherwise unreadable text. It has been documented across RL-trained reasoning models: R1, R1-Zero, and QwQ produce illegible chains of thought while their final answers stay readable (arXiv:2510.27338). It is now visible inside a frontier line too: Anthropic's Fable 5 / Mythos 5 system card reports and gives an extreme example of illegible reasoning, where a card-puzzle trace becomes incomprehensible.

If CoT can go unreadable under the training regime frontier models are moving toward, oversight needs a channel which does not depend on reading the model's words. Internal activations are one candidate: they sit downstream of the computation, not in the output the model controls. This project asks whether an activation-based monitor stays informative precisely where CoT becomes illegible.

Core question As CoT legibility drops, does a residual-stream activation monitor keep detecting what a CoT monitor can no longer read, and where does it fail?

Plan

1. Build an illegibility gradient. This is the main step. Construct a controlled set of reasoning traces that span fully legible to fully illegible, using two levers: prompts and tasks that push the model toward compressed or drifted reasoning, and models or checkpoints already known to differ in legibility. Fix an explicit legibility score for each trace so position on the gradient is measured, not eyeballed.
2. On the traces from that gradient, run two monitors targeting the same concept (for example, harmful intent): a CoT monitor that reads the reasoning text, and a residual-stream linear probe that reads internal activations.
3. Measure how each monitor's detection performance moves as legibility falls. The load-bearing output is the region where the CoT monitor breaks down and whether the probe still fires there.
4. Try to break the probe where it looks strong: obfuscation attempts, layer sweeps, threshold transfer across distributions.
5. Write up when the activation channel substitutes for an unreadable CoT and when it does not.

CoT monitoring is one of the few oversight tools that reads a model's own reasoning. Korbak et al. (2025) frame it as a valuable but fragile opportunity, and warn that shifts in training practice such as scaled-up RL threaten to degrade it. That erosion is already observed: across 14 reasoning models, RL often makes chains of thought illegible to humans and AI monitors while final answers stay readable (arXiv:2510.27338), and Anthropic's own Fable 5 / Mythos 5 system card reports a significant rise in illegible reasoning. If oversight is going to lean on CoT monitoring, we need to know whether a second, activation-based channel still works once CoT becomes unreadable, or whether both fail together. This project answers that on small open models, cheaply enough to run before the assumption becomes load-bearing in deployment. The result is decision-relevant either way: if the probe holds where CoT fails, it supports adding white-box monitors to the oversight stack; if it fails too, that warns against over-relying on either channel.

Mentees own a sub-question end to end: probe training and evaluation, illegibility-gradient construction or the obfuscation red-team. I set overall direction and the bar for what counts as a clean result, and expect mentees to run their own experiments, bring findings to weekly syncs, and push back when a baseline looks wrong. Early weeks are more guided (reproducing the base probe together); autonomy increases as shared infrastructure stabilises. Mentees keep a short log of experiments and negative results.

• High proficiency in Python.
• Have trained or fine-tuned a transformer in PyTorch (toy models or following guides is fine).
• Comfortable running experiments on a single GPU (Colab T4 level).
• Basic familiarity with transformer internals: residual stream, layers, activations. Prior probe experience is not required; comfort reading a paper and turning it into code is.
• Able to commit ~10 hr/week for the full round.

No geographic requirement. I ask for availability for one weekly sync and occasional debugging calls, flexible within European daytime and adjustable if the team spans wider time zones.