A Diagnostic Panel of Ground-Truth Probes for Language Models

Motivation

Safety monitoring today leans on probes and evals whose targets are borrowed from folk psychology. A deception probe is trained on labels that humans assigned, so every downstream claim inherits the label noise, the confounds, and the ambiguity of the concept itself. When such a probe fires on a new model, it is hard to say what, if anything, has been detected.

Medicine faced the same problem and solved it in a particular order: calibrated instruments first (temperature, blood pressure, lab panels), then syndromes discovered as reproducible clusters of readings, and only then the mapping to disease concepts. We propose the same order for language models. Start from probe targets where the ground truth is exact and computable from the output itself, so every claim stays falsifiable. Then look for regions of joint probe space that recur across seeds, models, and datasets, and let those define named conditions. Whether a discovered condition corresponds to a human concept like dishonesty becomes a downstream empirical question, not a modeling assumption.

What exists already

The first probe is built and published. Our NeurIPS 2026 submission (https://arxiv.org/abs/2607.05316) shows that a minimal linear probe on frozen hidden states predicts the remaining length $r\_t = T - t$ of a completion, that total length is decodable from the prompt's last hidden state before the first token is emitted, and that the estimate jumps upward at the exact token where the model says "Wait" and restarts a solution. Three models, seven datasets, and an evaluation cache that the panel work can reuse. A calibrated successor target is in progress: the log-survival probe predicts remaining cumulative hazard

which has unit-exponential marginals under the empirical length distribution, so probe outputs are automatically on the same scale across datasets and models.

Safety decisions about transformative AI will be made by reading numbers off models. Most of those numbers come from uncalibrated instruments: when a deception probe fails, we cannot tell whether the readout broke or the concept was never there. Ground-truth probes fix that. On targets like remaining length or end-of-sequence hazard the answer is exact, so failure is unambiguously a failure of method, which gives the field a validation floor. A probe pipeline that cannot recover what a model demonstrably knows about its own output has not earned trust about intent.

The Fall cohort will build outward from the length probe toward the panel. Concrete workstreams, sized so that each mentee can own one: train new ground-truth probes (end-of-sequence hazard, repetition onset, copy-versus-generate, format validity, a correctness probe on verifiable tasks); build jump and outlier diagnostics that flag completions where probe trajectories behave anomalously, then curate the datasets where those anomalies concentrate; and combine probes to search for the first candidate conditions. Two candidates are already specified and buildable from the existing cache plus two small probes: perseveration (the model idles, hazard stays low while repetition rises) and silent overrun (the model blows through its own predicted length without signaling). A discovered condition must pass four tests, in order: built from ground-truth-only probes, reproducible across seeds, models, and datasets, falsifiable in the sense that membership predicts a held-out verifiable outcome, and mechanistically coherent under patching. The honest risk, stated up front because it is also the interesting part: stable condition regions may not exist in joint probe space, and a careful negative result would itself be publishable.

All experiments run on open-weights models at the 7B to 8B scale, so the entry cost is a GPU-hours budget rather than lab access. Target output: a workshop paper or arXiv preprint per workstream, feeding a joint panel paper.

Core track: comfortable with Python and PyTorch, able to run inference and train small probes on open models; prior interpretability experience helps but is not required, since every target here comes with its own ground truth and its own sanity checks. A statistics background (survival analysis, calibration) is a plus for the hazard probes. Theory tracks: strong probability or decision theory, and interest in turning theorems into experiments. We care more about care in evaluation than prior safety experience.