Does Reinforcement Learning Improve a Transformer’s Access to Its Own Internal Errors?

This project asks a simple question: does reinforcement learning help a language model notice when something has gone wrong in its own reasoning, or does it only teach the model to give the response that earns reward?

The motivation is to understand computational properties that may be relevant to AI consciousness. There is no single accepted test of consciousness, so one approach is to study indicators suggested by different theories. This project focuses on metacognitive monitoring: the ability of a system to use information about its own internal processing to guide its next decision. More specifically, it studies internal-error monitoring—whether a model can detect that its current reasoning has become unreliable. This would not determine whether the model is conscious, but it could provide an additional indication of one consciousness-relevant capacity.

The experiment will compare 1–3B parameter Transformers that all begin from the same checkpoint. The main conditions will be an unchanged model, a model trained with supervised fine-tuning, a model trained with reinforcement learning, and a control model trained directly to report internal interventions. Keeping the starting model and training setup as similar as possible will help isolate what RL changes.

During short reasoning tasks, a hidden intervention will sometimes change the model’s internal activations without changing the visible prompt. Some interventions will disrupt the reasoning and make the model’s current answer incorrect; others will be equally large but harmless. Before giving its final answer, the model must choose either ANSWER or RETRY. A retry runs the problem again without the intervention but has a small cost. The best strategy is therefore to retry only when the model’s current reasoning has become unreliable.

Crucially, the RL model will never be rewarded for saying that an intervention occurred. It will receive reward only for reaching the correct final answer while avoiding unnecessary retries. This makes it possible to ask whether RL develops a useful internal-error signal without directly training the model to sound introspective.

The main questions are:

Does RL help the model retry specifically when its reasoning has been disrupted? Does this ability generalize to new interventions and new actions, or is it limited to choosing RETRY in the training task? When the model retries, is the decision guided by an internal signal representing the error, or has training simply made the output RETRY more likely?

The first stage will compare the RL model with the unchanged and supervised models on clean, harmful, and harmless intervention trials. The next stage will test new intervention types, layers, and strengths. It will also test whether the same information can guide behaviours that were not rewarded during training, such as abstaining, delegating a problem, comparing two internal states, or identifying where an intervention occurred.

The final stage will use mechanistic interpretability methods to investigate why the model retries. A classifier will search for an internal pattern that predicts when the reasoning has been disrupted. The experiment will then remove or copy this pattern to test whether it causes the retry decision. A separate intervention near the model’s output will directly increase the probability of RETRY, providing a control for behaviour that looks like self-monitoring but is produced only by changing the output.

The main contribution would be a controlled test of whether RL strengthens a reusable form of internal-error monitoring or mainly changes a model’s response policy. The result could also matter for AI safety: a reliable internal-error signal could help future systems request recomputation or oversight when their reasoning becomes unreliable, while a narrow or easily steered response could make model self-reports appear more trustworthy than they are.

The initial scope is one open model family, one reasoning-task family, and one intervention method. Extensions could include additional model sizes, intervention types, or deeper analysis of the learned mechanism.

Relevant papers:

Consciousness in Artificial Intelligence: Insights from the Science of Consciousness — Patrick Butlin et al.

Indications of Belief-Guided Agency and Meta-Cognitive Monitoring in Large Language Models — Noam Steinmetz Yalon, Ariel Goldstein, Liad Mudrik, and Mor Geva

Emergent Introspective Awareness in Large Language Models — Jack Lindsey

Mechanisms of Introspective Awareness — Uzay Macar et al.

Introspection Fine-Tuning: Training Small LLMs to Introspect — Ely Hahami, Ishaan Sinha, and Lavik Jain

AI systems may need to recognize when their own reasoning has become unreliable and request recomputation or human oversight before acting. This project tests whether reinforcement learning can create a reusable internal-error monitor, rather than merely training models to produce reassuring self-reports. The results could inform scalable oversight and deployment safeguards by showing when model self-reports provide meaningful evidence, and when relying on them may create false confidence.

Initial 1:1, then weekly meetings and additional adhoc 1:1’s

Strong Python skills and experience with PyTorch. Familiarity with Transformer language models and ideally the Hugging Face ecosystem. Basic understanding of reinforcement learning or experience fine-tuning language models. Be comfortable doing conceptual thinking on their own

Mentees should be available for check-ins during some one-to-two hour slot within 9am - 6pm Central European Time.