Towards Automated Vulnerability Discovery and Repair with Safety-Governed AI Agents

AI agents are rapidly improving at code understanding, tool use, and long-horizon software engineering. These capabilities create an opportunity to automate parts of vulnerability discovery, validation, and repair, but current systems remain difficult to trust in real security workflows. A model may generate plausible but incorrect findings, overlook cross-file attack paths, misuse powerful tools, or produce patches that remove a symptom without fixing the root cause. More importantly, the expert data needed to train and evaluate reliable vulnerability-research agents is scarce: most public datasets contain final vulnerability labels or patches, rather than the full process by which security experts form hypotheses, inspect evidence, reject false positives, reproduce vulnerabilities, and validate repairs.

This project investigates two pieces of foundational infrastructure for future automated vulnerability discovery and repair.

Workstream 1: Expert-in-the-loop Safety Harness. We will prototype a lightweight research environment in which security experts can collaborate with AI agents while retaining control over the investigation. The harness will support repository exploration, model and tool orchestration, isolated execution, evidence tracking, expert approval, and reproducible replay. Rather than treating the model's first answer as a finding, the workflow will encourage a structured sequence: understand the target, propose a vulnerability hypothesis, gather code and tool evidence, attempt validation in a sandbox, obtain expert review, and then generate and test a repair. The harness will enforce a structured workflow:

“Target understanding → vulnerability hypothesis → evidence collection → sandbox validation → expert review → patch generation → repair testing”

The design will draw inspiration from emerging multi-model scanning systems such as Microsoft's MDASH, while focusing on expert supervision, transparent evidence, and research data collection rather than large-scale autonomous scanning.

Workstream 2: Security Data Engine. The harness will be instrumented to capture structured vulnerability-research trajectories. These trajectories may include repository context, files inspected, hypotheses proposed, security-tool calls, intermediate evidence, failed validation attempts, expert corrections, false-positive rationales, patch revisions, and regression-test outcomes. The data engine will convert these heterogeneous interactions into auditable, privacy-aware records that can support supervised fine-tuning, preference learning, tool-use optimization, workflow evaluation, and benchmark construction. Failed and rejected trajectories are particularly important because they reveal where agents appear convincing but are wrong.

The central research questions are:

1. What interaction design makes AI assistance genuinely useful to security experts without reducing expert control or creating excessive review burden?
2. Which parts of expert vulnerability-research trajectories are most informative for improving agent reliability, validation accuracy, and repair quality?
3. Can structured expert feedback and replayable evidence reduce false positives and improve the reproducibility of AI-generated findings?
4. How should high-risk cyber-agent capabilities be constrained, logged, and evaluated so that research progress does not come at the cost of unsafe deployment?

We will begin with a deliberately scoped prototype rather than attempting to build a full cybersecurity foundation model. Initial experiments will use authorized open-source repositories, known vulnerabilities or seeded security defects, and a small set of established analysis and testing tools. Mentees will help implement the harness and trajectory schema, run pilot studies with security practitioners, and compare model-only workflows against expert-supervised workflows. Evaluation will consider finding validity, evidence completeness, reproduction success, false-positive burden, patch correctness, expert time, and the usefulness of collected trajectories for improving subsequent agent runs.

Expected outputs include: (i) an open research prototype of the expert-in-the-loop safety harness; (ii) a structured schema and data pipeline for vulnerability discovery and repair trajectories; (iii) a small, carefully governed dataset of expert-reviewed trajectories where release is legally and ethically appropriate; and (iv) an empirical study of how expert feedback and safety constraints affect AI-assisted vulnerability research. The longer-term goal is to create a continuous data flywheel in which expert use produces better data, better data improves agents, and improved agents return to the harness for further supervised use.

Relevant references: Microsoft MDASH: https://www.microsoft.com/en-us/security/blog/2026/05/12/defense-at-ai-speed-microsofts-new-multi-model-agentic-security-system-tops-leading-industry-benchmark/ AgentHazard: https://arxiv.org/abs/2604.02947 BraveGuard: https://arxiv.org/abs/2606.01166

Advanced AI agents will increasingly be able to inspect large codebases, invoke security tools, reproduce vulnerabilities, and generate repairs. These capabilities can strengthen cyber defense, but they are also dual-use: unreliable agents may create false confidence, damage systems through unsafe tool use, or lower the barrier to offensive cyber activity. Safe progress therefore requires more than increasing model capability. It requires mechanisms for expert oversight, constrained execution, evidence-based validation, auditability, and responsible data governance.

This project develops these safeguards alongside capability research. For example, if an agent claims that a repository contains a command-injection vulnerability, the safety harness will require it to identify the relevant code path, provide evidence, reproduce the issue in an isolated sandbox, and obtain expert approval before testing a patch. High-risk actions, such as running exploit code, accessing external networks, or modifying repositories, will be permission-gated, logged, and replayable. If the claim cannot be reproduced, or the patch breaks functionality, the failed attempt and expert correction will also be retained.

The theory of change is: expert-supervised use produces high-quality behavioral data; this data enables better safety training and evaluation; improved agents are redeployed only within an auditable harness; and repeated supervised use creates a feedback loop that improves defensive usefulness while preserving human control. The project is related to our work on trajectory-level safety monitoring for computer-use agents, AgentHazard: https://arxiv.org/abs/2604.02947 and BraveGuard: https://arxiv.org/abs/2606.01166.

Mentees will work as research collaborators with substantial ownership over a clearly scoped component, while receiving weekly guidance and code/research review. Depending on background and interest, a mentee may focus on one of two tracks.

Harness track: Design and implement a lightweight expert-facing workflow for repository analysis, model/tool orchestration, sandboxed validation, evidence capture, and expert approval. The goal is not to reproduce a production platform, but to build a usable prototype that supports controlled experiments.

Data-engine track: Design a structured trajectory schema; implement logging, normalization, replay, and annotation components; and study how expert corrections, rejected findings, and failed validation attempts should be represented for later training and evaluation.

All mentees will contribute to literature review, experimental design, pilot evaluations, and research writing. They will be expected to propose design choices, maintain reproducible experiments, document limitations, and present progress in weekly meetings. Mentees will have autonomy over implementation and exploratory experiments after agreeing on a milestone plan. High-risk testing must use authorized targets and isolated environments, and any potential real-world vulnerability will follow coordinated responsible-disclosure procedures.

Applicants should have:

• Strong Python programming skills and experience building, integrating, or evaluating LLM/agent systems, software-engineering tools, or security-analysis pipelines.

• Solid foundations in either machine learning/AI systems or cybersecurity/software engineering.

• The ability to work independently on open-ended research problems, read and implement research papers, debug complex systems, and communicate results clearly.

• First-author publications or substantial contributions to published work will be considered particularly valuable.

Experience in at least one of the following areas is strongly preferred:

• AI security, vulnerability research, program analysis, secure coding, fuzzing, automated program repair, or CTF-style security problem solving.
• Coding agents, tool-using LLMs, agent evaluation, model fine-tuning, preference learning, reinforcement learning, or trajectory-data construction.
• Building substantial research prototypes, open-source systems, security tools, technical reports, or reproducible experimental pipelines.

Applicants do not need to be experts in both AI and cybersecurity, but they should be strong in at least one area and able to learn the other quickly. Clear technical writing, responsible handling of security-sensitive work, and evidence of completing research projects are essential.

No geographical restriction. The team will meet online once per week. Mentees should be able to attend a recurring meeting during a mutually agreed window with reasonable overlap with Singapore time (UTC+8), and communicate asynchronously through shared documents and a code repository.