Locating the Refusal Circuit in Unified, Encoder-Free Vision-Language Models

Background. Vision-Language Models reliably refuse harmful text prompts but frequently comply with the same harmful intent when delivered visually, a gap demonstrated across benchmarks like FigStep, MM-SafetyBench, and HADES. This matters beyond benchmarks: VLMs are already the perception/decision backbone in computer-use agents, autonomous driving stacks, and robotic manipulation policies, all of which inherit whatever safety properties (or failures) exist in the underlying model's visual channel. Our own prior work, VLM-RobustBench (Saxena, Suglia, and Minervini; ICML 2026), shows this channel is already fragile even without adversarial intent. For instance, low-severity, non-adversarial visual perturbations can degrade VLM behaviour more than visually severe ones, which motivates taking deliberately crafted visual jailbreaks seriously as a security problem, not just a benchmark curiosity. In modular VLMs (a frozen vision encoder + lightweight projector feeding an LLM backbone), this has been mechanistically localised: safety training concentrates refusal into a single linear direction in the LLM's residual stream (Arditi et al. 2024), but the projector, trained only on benign image-text pairs, never learns to map toxic visual concepts into that direction. Multiple 2025/2026 papers (cross-modal transfer, ShiftDC, CARE, DTR) have already localised and partially patched this projector-LLM interface for modular architectures.

The open question. Unified, encoder-free VLMs (e.g. Gemma-4-class models: google/gemma-4-12B-it) eliminate the discrete vision tower and projector entirely, projecting raw image/audio patches straight into a shared decoder via linear layers. If these models still exhibit visual jailbreak vulnerability, there's no obvious module to blame — the failure must live somewhere else: distributed across early decoder layers, tied to positional encoding of patch tokens, or emergent from training-data composition. No published work has addressed this yet, and past work suggests architecture may play an important role in how information flows through refusal circuits (Marbut et al. 2026, Expert-Aware Refusal Steering) and in influencing the geometrical properties of refusal cones (Wollschlager et al. 2025, The Geometry of Refusal in Large Language Models). Therefore, it's fundamental to study and understand how such a novel architecture may be affected by unsafe and harmful behaviours to ensure a safe and secure deployment.

Research questions. (1) Do encoder-free VLMs show the same magnitude of text-to-vision refusal gap as modular VLMs, evaluated under leakless conditions (VLSBench, which decouples visual safety information from text so refusal can't shortcut off the text channel)?

(2) If the gap exists, where does it live without a projector to serve as a natural probing boundary?

(3) Does a single linear "refusal direction" still exist cleanly in a unified decoder, or does removing the projector bottleneck change how refusal is represented?

(4) Does the "hydra effect" — redundant refusal circuits reactivating after a primary component is ablated — behave differently in unified vs. modular architectures?

The research team that will be working with you on this project is:

• Dr Alessandro Suglia (Lecturer in Embodied NLP at the University of Edinburgh)
• Dr Francesco Pinto (Research Scientist at Google DeepMind)
• Dr Rohit Saxena (Postdoc at the University of Edinburgh)
• Sophie Higham (PhD student at the University of Edinburgh)

Please refer to the attached document for additional background information and a potential work plan.

As VLMs move toward encoder-free, unified architectures for efficiency and on-device deployment (smaller footprint, single fine-tuning pass, lower latency), this architecture class is likely to become a default deployment target rather than a niche — including in agentic and consumer settings where multimodal jailbreaks and prompt injections translate directly into real-world harm. This is not hypothetical: VLMs already autonomously invoke APIs that have privileged access to users' personal accounts (e.g., banking, email, etc.), drive computer-use agents that click and type on real machines, autonomous-driving perception/planning stacks, and robotic manipulation policies (e.g. OpenVLA) — all inheriting whatever safety failures exist in the underlying model's visual channel. Our own VLM-RobustBench work shows this channel is already fragile even under mild, non-adversarial corruption, which only strengthens the case for taking deliberate visual jailbreaks seriously. Existing multimodal safety interventions (VLGuard, ECSO, AdaShield, ShiftDC, CARE, DTR) all assume a discrete projector to target; none of them straightforwardly transfer to a unified architecture. Understanding safety failures in this new type of models is fundamental to prevent their future deployment with a possibly inherently different class of vulnerabilities that require novel defences, compared to the existing ones focused on modular VLMs only. This project produces (a) an early empirical read on whether the modality safety gap generalises to unified architectures, (b) a first mechanistic account of where it lives without the usual projector target, and (c) a proof-of-concept intervention — laying groundwork other researchers and labs can build on before unified architectures are ubiquitous.

Mentees will be part of the Embodied, Situated and Grounded Intelligence group at the University of Edinburgh (https://alesuglia.github.io/esgi/). They will therefore benefit from exposure to a variety of different topics in the broader Multimodal and Embodied AI space as well as the broader School of Informatics at the University of Edinburgh, which hosts one of the best AI/NLP departments in the world.

Mentees will own specific experimental threads under weekly direction: e.g. in month 1, both mentees set up the experimental codebase and baselines to have a robust and reproducible codebase. Then, Mentee 1 leads the difference-of-means refusal direction while Mentee 2 uses logit-lens tracing. In the final month, both mentees collaborate to design and evaluate a targeted intervention benchmark safety/utility tradeoffs on VLSBench + MMBench/MMLU against best-effort modular baselines, and write up for a technical report (e.g., https://www.lesswrong.com/). Mentees are expected to propose and run their own experiments within the agreed weekly plan, make implementation decisions independently (e.g. exact hook placement, patching methodology), and flag blockers early — this is a genuinely open research question, not a pre-specified engineering task, so initiative in interpreting ambiguous results matters as much as coding ability. I'll set direction and sanity-check findings in weekly syncs, review code/experiment design, and step in directly when mentees are blocked on the newer, less-documented parts of the model's architecture.

• Highly proficient in Python and comfortable in PyTorch and HuggingFace Transformers, including loading and modifying open-weight model code (not just calling high-level pipelines).

• Comfortable writing manual PyTorch forward hooks to extract or modify intermediate activations, without relying on a pre-built library — TransformerLens-style tooling won't have out-of-the-box support for this model's architecture, since it's only weeks old.

• Knowledge of Vision-Language-Models or experience in using them, going beyond prompting via API.

• Bonus (not required): prior exposure to interpretability techniques (SAEs, activation steering, logit lens, causal mediation), LLM safety/red-teaming.

• Bonus (not required): experience with cloud GPU rental workflows (RunPod, Lambda, Vast.ai, Lightning AI) rather than only managed notebook environments.

• Bonus (not required): experience with logging and reporting tools such as W&B

No strict geographic requirement. Preference for enough timezone overlap to hold a consistent weekly sync (The PI and his collaborators are based in the UK).