Transcript Analysis for Cybersecurity Evals

The SaferAI risk modelling team is looking for new inputs to our risk models given our view that benchmarks are weak (and rapidly saturating) proxies for operational capability in live, defended cyber environments. One input we are considering is using open ended evaluations (like cyber ranges) or real-world red team data.

In order to make use of these kinds of evaluations in risk models, we need to figure out how to process their outputs - they are generally much less structured than benchmarks. Transcript analysis could be a good option for this, and seems to be a direction some folks in the field are moving towards. It also could easily be repurposed for agentic benchmark runs (like CyBench), or more structured cyber ranges which could extend the utility of already saturated benchmarks.

One of the important functions of a risk model is to provide a substrate for integrating many types of data, drawing operational conclusions by integrating information from diverse sources into a common quantitative “language”. We have not yet explored the potential of risk models to use any sorts of data beyond benchmarks, and transcripts are a natural first step in that they are relatively plentiful, generally applicable, easily adapted to reflect red team data beyond evals data, and provide higher resolution signal than a binary “pass / fail” metric.

We are one of less than 5 organizations with a fully functional evals team which can provide cyber-relevant eval transcripts as needed. In addition, we have cybersecurity partners who can provide live-environment transcripts as well. Taking advantage of this data for this kind of work is high-impact.

SaferAI’s Risk Modelling Theory of Change: We develop models with the goal of identifying rapidly advancing and/or highly consequential offensive capabilities.

Project Goal: Current risk models don’t have enough data to resolve detailed questions about capability with high uncertainty, so we need to find ways to improve our input data. Transcripts are a good option, and will improve our understanding of capabilities, and be useful to the AI Safety field at large.

Audience(s): Policy Makers and Operational Security Community, potential extension to AI Safety researchers

What it does for Policymakers: Outputs can better direct funding towards research areas which counteract identified developing capabilities. This could result in updated vulnerability disclosure practices, patching requirements for government/critical infra, use of open source projects, etc

What it does for Operational Security Community: This project provides insight into emerging dynamics and how to adapt defense strategies (would be ideal to partner with security orgs for these kinds of recommendations). Private defense market has massive incentives to develop defensive capabilities, but likely need better signal on where to direct this funding

AI Safety Researchers: Many evals are emerging out with a common theme, “how do we make this make sense to decision makers”. Transcript analysis provides a rich interface between eval runs and real-world impact, and opens up the eval design space by making more evals translatable into legible impact.

See proposal for finer details, but broadly the mentee will:

Design an annotation schema and extraction pipeline for transcript analysis (informed by Inspect Scout and recent log-analysis literature).

Run the pipeline on CyBench transcripts, then on live red-team data from SaferAI's cybersecurity partners, eventually mapping extracted metrics onto risk-factor parameters in SaferAI's existing risk models.

Draft a methodology paper and an applied blog post analyzing results.

Comfort with LLM eval transcripts / agentic tool-use logs

Significant Python experience; familiarity with Inspect and Inspect Scout

Cybersecurity background helpful but not required