Guarding Against Malicious Fine-Tuning: Hardening Models and Detecting Tampering

Fine-tuning interfaces allow users to adapt powerful language models to specific domains, but they also present a serious security risk: determined adversaries can use fine-tuning to override safety training, insert backdoors, or create models that behave benignly in most cases while producing dangerous outputs in response to special triggers. Recent work on covert malicious finetuning and instruction-based backdoor attacks demonstrates that attackers can construct fine-tuning datasets where each example appears innocuous, yet the resulting model reliably performs harmful behaviours while evading standard safety checks.

This project has two main components:

1. Hardening against malicious fine-tuning: We will experiment with training-time and fine-tuning-time defences that make models more resistant to being repurposed for harmful behaviour. Examples include: partially freezing safety-critical layers; adding regularisers that penalise large weight changes in specific subsystems; incorporating adversarial training using simulated backdoor attacks; and exploring lightweight watermarking or fingerprinting techniques that make it easier to verify model integrity after deployment.
2. Tamper forensics and detection: We will develop methods to detect whether a model has been maliciously fine-tuned. Approaches include: behavioural probes (carefully designed prompt suites that elicit backdoor behaviour, if present); weight-space comparison between a suspect model and a trusted reference checkpoint; and auxiliary ‘monitor’ models that evaluate responses for signs of hidden policies. We will build small benchmarks of synthetic attacks—e.g., trigger-phrase backdoors, policy-bypassing adaptations—and measure how well different detection strategies identify compromised models.

The outcome will be an empirical characterisation of how vulnerable current LLMs are to various malicious fine-tuning strategies, plus baseline defences and detection tools that could be used by labs or auditors to monitor real-world models.

Even if a foundation model is trained to be safe, its alignment can be undermined post-hoc if adversaries can secretly fine-tune it for harmful purposes. For transformative AI systems, this poses a major governance and deployment risk: a seemingly benign model could, in fact, be a tampered system with hidden dangerous capabilities. By hardening models against malicious fine-tuning and developing practical detection methods, this project helps ensure that safety properties are preserved across the full lifecycle of a model, not just at pretraining.

The theory of change is twofold. First, technical hardening increases the cost and difficulty of successful attacks, making it less likely that low-effort adversaries can create undetected rogue models. Second, reliable detection tools allow labs, regulators, or downstream users to verify that a deployed model matches its audited version, and to flag or quarantine suspect models. Together, these measures reduce the risk that powerful AI systems are quietly repurposed for misuse, and complement other alignment and governance efforts.

Mentees will take on core responsibilities in both the attack and defence sides of the project, under supervision but with significant freedom to shape experiments. Likely tasks include:

• Implementing and running malicious fine-tuning attacks on open-source LLMs (e.g., trigger-based backdoors, policy evasion via instruction attacks).
• Designing and evaluating training-time defences (such as weight freezing, regularisation schemes, or adversarial fine-tuning) and measuring their effect on both task performance and attack success.
• Developing and benchmarking detection pipelines that compare suspect models to trusted baselines via behavioural tests and weight-space analysis.
• Performing rigorous quantitative analysis of attack and defence effectiveness, and contributing to written summaries or a manuscript.

Mentees will be encouraged to propose their own attack variants or defence ideas, and to iterate on them in response to empirical results.

• Strong proficiency in Python.
• Hands-on experience training or fine-tuning neural networks (ideally Transformers) using PyTorch or TensorFlow.
• Familiarity with large language models and basic prompt engineering; prior use of HuggingFace or similar tooling is highly desirable.
• Comfort with experimental design and empirical evaluation (e.g., setting up baselines, tracking metrics, and debugging ML pipelines).
• Interest in security, robustness, or AI safety; willingness to think adversarially about model behaviour. Optional but beneficial: prior exposure to adversarial attacks, backdoors, or security in ML systems.

No