Benchmark Cartography: Using Interpretability to Map What Our Evaluations Miss

When a benchmark saturates, the field's response is to build a harder one, and the implicit assumption is that harder means better. But difficulty is not validity. A benchmark can become more difficult while covering a narrower slice of its domain, and nothing in the current workflow would catch that. We have rigorous methods for evaluating models and almost none for evaluating the instruments we evaluate them with.

This project treats benchmarks as measurement instruments and asks whether the instruments are improving. The method builds on two lines of prior work:

• SAE-based competency gaps, which decomposes model activations into SAE concepts and computes saliency-weighted performance across benchmark items, producing two outputs at once: where a model is weak, and where a benchmark fails to cover.
• SAE-based blindspot identification, which shows that generative models have systematic conceptual gaps of two kinds—suppressed, where a concept is severely under-represented, and exaggerated, where one is over-emphasized—and that exposure during training does not imply capability.

The transfer is direct: benchmarks have blind spots the same way models do. A benchmark can under-probe a concept region or over-index on one, and containing items about a concept does not mean it measures competency in it. Competency Gaps currently produces this as a static, single-benchmark diagnostic. This project takes it in two new directions: longitudinal, applying it across benchmark generations, and generative, using identified gaps to construct items that probe what existing evaluations miss.

RESEARCH DIRECTIONS

A. Longitudinal benchmark cartography. Take a benchmark lineage in one domain, embed items into SAE concept space, and measure coverage breadth, concentration, redundancy, and generation-over-generation overlap. Does each successive benchmark expand the covered region, or increase difficulty inside the same footprint? Output: coverage maps over time for at least one lineage, ideally two from different domains, so we can say something about whether the pattern generalizes.

B. Suppressed and exaggerated benchmark blind spots. Port the blind-spot taxonomy from generative models to evaluations. Which concept regions are systematically under-probed across a domain's benchmarks, and which are over-weighted? Then the sharper question: are the blind spots correlated across benchmarks that were built independently?

C. Interpretability-guided item construction. Generate items targeting under-covered regions and validate them: do models that saturate the parent benchmark fail the new items? If they do, the parent was overstating competency, and we have a constructive method rather than only a critique.

D. Disentangling model progress from benchmark drift. A two-way design: hold the benchmark fixed and vary the model generation, then hold the model fixed and vary the benchmark generation. Cross-tabulate both against a concept-space competency measure that does not move when the benchmark does.

THE HARD PART, STATED UP FRONT

If we use a model's SAE features as the coordinate system for judging benchmark coverage, we are measuring coverage relative to that model's concept space. A benchmark cannot fairly be faulted for missing concepts the reference model does not represent. This is the central methodological risk and the project is designed around it: we use several models' feature spaces and report agreement, treat disagreement regions as their own category, and state claims as coverage relative to a representational frame rather than ground-truth domain coverage. A mentee who finds this problem interesting rather than discouraging is the right fit.

A second confound worth naming: harder items may naturally live in different concept regions, so "expanded coverage" and "increased difficulty" can entangle. Thread A includes a difficulty-matched control to separate them.

RELATION TO EXISTING WORK

This sits in conversation with measurement-validity and psychometrics literature: The Benchmark Lottery (Dehghani et al.), AI and the Everything in the Whole Wide World Benchmark (Raji et al.), Measurement and Fairness (Jacobs and Wallach), item response theory applied to LLM evaluation (tinyBenchmarks, Polo et al.), and benchmark-error audits such as Are We Done with MMLU? (Gema et al.).

The differentiator: IRT-style approaches work from response patterns only, so they can tell you that an item discriminates but not what it discriminates on. Working in concept space gives semantic content to the gaps, which is what makes item generation possible at all.

GOALS AND OUTPUTS

• A paper targeting a main track or a datasets-and-benchmarks track, presenting coverage cartography across at least one benchmark lineage plus the shared-blind-spot analysis. Mentee co-authorship for substantive contributions.
• An open tool that takes a benchmark and a model and returns a concept-space coverage map, reusable by anyone building an evaluation.
• Released generated items from thread C with validation results.

Scope is deliberately larger than three months of work. Part of my job is choosing which threads to cut once we see early results.

Dangerous-capability evaluations, refusal evaluations, alignment and deception evaluations are all benchmarks. "The model does not do X" always means "the model did not do X on the items we wrote." If safety evaluations have systematic conceptual blind spots, unsafe behavior survives testing in precisely those regions, and we would have no signal that it happened.

The sharpest version of this concerns shared blind spots. Benchmarks in a domain are built by overlapping communities drawing on similar intuitions about what matters. If nominally independent evaluations miss the same concept regions, then the apparent robustness of a capability or safety claim across several benchmarks is an illusion, and the field's core defense against measurement error, triangulation across instruments, does not work. Establishing whether this is true seems more valuable than building another benchmark.

Evaluation thresholds are also load-bearing for governance. Frontier safety frameworks trigger deployment and mitigation decisions on evaluation results. That makes construct validity a governance problem, not just a matter of research hygiene: a threshold is only as meaningful as the instrument that measures it.

Finally, saturation is ambiguous in a way that matters for forecasting. When scores hit ceiling, the scores alone cannot tell us whether models genuinely acquired the competency or whether the benchmark stopped discriminating. Separating those requires an instrument that does not live inside the benchmark. Model internals are that instrument, which is the cleanest argument for why interpretability belongs in evaluation science at all.

The constructive half matters as much as the critical half. If gap-guided item construction works, it gives the field a repeatable method for building evaluations that target what current ones miss, instead of iterating on difficulty and hoping coverage follows.

Each mentee owns a primary research thread (though, much of the underlying infrastructure, data, and analysis is shared). I am prepared to define the RQ and provide the literature scaffold with starting code --- though, I expect that we will start with a conversation, all of these constraints will be open to debate (reflecting interest and skill). The mentee owns experimental design, implementation, analysis, and the writeup of their section, with weekly review and discussion.

The first two weeks are a shared onramp: reproduce a concept-space coverage map for a single benchmark, so everyone has the tooling working and we share a common reference before threads diverge. After that, autonomy increases quickly. I expect mentees to run experiments between syncs and arrive with results and blockers rather than waiting for direction.

Thread A is the most analysis-design heavy and suits someone with strong data instincts. Thread B is the most conceptually open. Thread C involves item writing and validation and suits someone who thinks adversarially about what a question actually measures. I will match threads to people after the first two weeks, not before.

Substantive contributors get authorship.

Required:

• High proficiency in Python and PyTorch, and comfort running inference over open-weight models. No SAE training is required; pretrained SAEs (Gemma Scope, Llama Scope) cover the models we would use.
• Strong data analysis and visualization skills. A large share of this project is analysis design rather than engineering: deciding what to measure and how to show it is the work.
• Ability to read a benchmark critically. You should be able to look at a dataset and articulate what it does and does not measure, beyond reporting its stated purpose.

Useful, not required:

• Familiarity with sparse autoencoders or feature-level interpretability. If you have not used them, demonstrated ability to pick up an unfamiliar toolchain quickly is an acceptable substitute.
• A measurement, psychometrics, or statistics background. This is unusual in ML and a genuine asset here. If you have thought about construct validity, item discrimination, or test design in any domain, including outside ML, say so in your application.
• For the item-construction thread, experience writing careful test items or exam questions.