Can We Trust the Failure Detectors? A Validity Audit of Trace-Based Monitoring for LLM Agents

The question. Agent monitoring today runs on failure detectors: trace-signal heuristics, LLM judges, and hybrids. How valid are they? Not how consistent, how valid: do they recover the judgments careful humans make about the same traces? We will answer this with a preregistered validity audit against double-annotated human ground truth, and release the labeled corpus openly.

Why now. A recent audit of 21 LLM-judge implementations across 9 providers (https://arxiv.org/abs/2606.19544) found that standard exact-match validation overstates judge ability: agreement deflates by 33–41 percentage points once chance-corrected (Cohen's kappa on MT-Bench), and two production-deployed judges combined test-retest reliability above 0.95 with position bias. They are consistent but not valid. Human-annotated agent-trace resources exist: TRAIL (https://arxiv.org/abs/2505.08638) annotated 148 GAIA/SWE-bench traces against a fine-grained error taxonomy, and MAST (https://arxiv.org/abs/2503.13657) annotated 150+ multi-agent traces with reported inter-annotator agreement. What none of them provides is a validity audit of the deployed detector classes themselves, with chance-corrected per-failure-class statistics, preregistration, blinding, and deliberate oversampling of safety-salient failure classes (unsafe tool use, guardrail bypass, specification-gaming-shaped behavior). That audit, plus the corpus it produces, is this project.

Integrity by construction. I am the sole author of AgentTelemetry (AIware 2026, ACM, https://doi.org/10.1145/3805760.3814931), a fault-detection benchmark whose LLM-judge-derived labels were flagged in peer review as a validity limitation. This project applies the cure to my own prior work, and the design keeps me out of the loop twice over: I never produce or adjudicate a label, and I never pick the test set. A candidate trace pool with per-trace provenance is collected under written inclusion rules and committed publicly before preregistration; a mentee draws the stratified study sample by executing the preregistered selection rules; annotators are blind to detector outputs and injection status; judge prompts and detector configurations are frozen at preregistration.

Plan (12 weeks, 3 mentees). Weeks 1–3: stress-test the pre-built codebook (derived from MAST failure modes and TRAIL error types, cross-checked against a production-incident catalog, https://arxiv.org/abs/2606.04056), annotate a ~30-trace pilot, and pass a hard gate: Krippendorff's alpha ≥ 0.6 within two codebook revisions, else collapse to a coarser taxonomy. Preregister on OSF. Weeks 3–8: label 200–300 length-stratified traces, of which ≥60–70% are natural traces from public benchmark rollouts (SWE-bench leaderboard trajectories, GAIA, WebArena/OSWorld) plus a clearly-marked synthetic minority stratum; a stratified ~50% core is double-annotated; a week-6 checkpoint applies a preregistered descope rule. Weeks 8–12: run the detectors (trace heuristics including AgentTelemetry's, 2 LLM-judge configurations, and a hybrid) against the human labels; preregistered confirmatory statistics (per-label alpha, prevalence, bootstrap CIs, overall plus the most prevalent classes) with the rest exploratory; write the paper. Mentees draft, I edit.

Operating model. This project is designed to run on artifacts, not on mentor presence: codebook v0, the candidate corpus, the annotation interface, harness and analysis skeletons, and a written charter (weekly goals, escalation rules, gates, authorship policy) are built before the round starts (public repo: [https://github.com/Krishnachaitanyakc/agent-monitor-validity]). Each mentee owns a workstream and its paper section: codebook and statistics; tooling and detectors; data and release. One mentee serves as project lead. I engage mostly asynchronously: written feedback on weekly updates within 72 hours, biweekly office hours, reserved gate decisions, and hands-on editing in the final two weeks.

Outputs. (1) A paper, submission-ready by week 12, targeting the ICLR 2027 workshop cycle or an SE data/benchmark track; mentee first authorship by default, order by contribution, mentor last. (2) The human-labeled agent-failure corpus, released openly (code public from day one; trace data after a documented license/ToS/PII scrub; benchmark rollouts only, never real-user traces). (3) Possible follow-on contribution to a public eval registry (e.g., UK AISI inspect_evals).

Automated failure monitors are becoming the load-bearing component of how labs and deployers claim agentic systems are safe: AI-control-style safeguards assume unsafe behavior will be caught by monitors, and safety evaluations lean on LLM judges to score agent behavior at scale. If monitor validity is unmeasured, every safety case built on monitoring inherits an unquantified error term, and there is direct evidence this term is large (https://arxiv.org/abs/2606.19544: 33–41 percentage points of agreement deflation once chance-corrected, with production judges that are highly consistent yet biased). The failure mode is false assurance: oversight that reliably beeps for burnt toast and sleeps through fires, certified by counting how often it beeps.

This project attacks the problem at its root by producing (1) reliability-quantified, double-annotated ground truth built specifically to audit deployed detector classes, with safety-salient failure classes (unsafe tool use, guardrail bypass, specification-gaming-shaped behavior) oversampled and reported per class, and (2) an honest, preregistered, blinded measurement of how much trust each detector class deserves. Both are released openly so monitor developers, control-protocol designers, and evaluators can calibrate their reliance on automated oversight and improve against human-grounded labels rather than judge-derived ones. It also corrects a flagged validity limitation in my own published benchmark (AgentTelemetry, AIware 2026, https://doi.org/10.1145/3805760.3814931), which is the standard of self-correction the field's measurement layer needs. Related completed work: the AgentTelemetry benchmark and the agent-planning telemetry conventions I authored in OpenTelemetry GenAI, the observability layer such monitors consume.

High autonomy by design. Mentees own the study: each of the three mentees leads a workstream (codebook and statistics; tooling and detectors; data and release) and owns the matching paper section, and one mentee serves as project lead (nominated at the week-1 kickoff, confirmed by me). The study sample is drawn by a mentee under preregistered rules; labels are produced and adjudicated entirely by mentees; the confirmatory analysis is run by mentees on a provided analysis skeleton. I provide direction and scaffolding rather than day-to-day management: written feedback on weekly updates within 72 hours, biweekly office hours, decisions on the charter's reserved gates (codebook collapse, week-6 descope, submission target), and hands-on paper editing in weeks 11–12. First authorship goes to a mentee by default, expected to be whoever drives the confirmatory analysis; I take the last-author role. Annotation is capped at roughly half of each mentee's weekly hours so their ownership work is protected.

Required:

1. High proficiency in Python (you can build and debug small tools without hand-holding).
2. Basic statistics: comfortable with the idea of chance-corrected agreement; Krippendorff's alpha itself is learnable from the provided analysis skeleton.
3. Demonstrated self-directed work: a prior research project, a substantial open-source contribution, or equivalent evidence that you can drive a workstream on asynchronous feedback. This project deliberately offers autonomy rather than close supervision, so this is the prerequisite that matters most.
4. The patience to do careful annotation as part of the work (capped at about half your hours, but it is real and must be done well).
5. 8–10 hours per week for 12 weeks, plus availability for the biweekly office-hours call.

Nice to have (not required): experience with LLM agent frameworks; observability/OpenTelemetry; prior annotation-protocol, dataset, or ML-evaluation work.

No geographical preference; the project is async-first. Required availability: a recurring biweekly 1-hour office-hours slot and a one-time week-1 kickoff call, in a window of roughly 5–7pm US Pacific on weekdays or 9–11am US Pacific on weekends. Everything else is asynchronous on Slack/GitHub.