Narrow finetuning can install broad unintended behaviours, e.g., emergent misalignment, subliminal learning, and inductive backdoors, through data that looks harmless or unrelated. This project asks whether these behaviours are carried by sparse subnetworks, and whether those subnetworks pre-exist in the base model or are created by the finetune, through the lens of the lottery ticket hypothesis.
About the project
The lottery ticket hypothesis (Frankle and Carbin, 2018, Renda et al. 2020,) holds that dense networks contain sparse subnetworks capable of expressing particular functions. Applied to unintended generalization (Betley et al. 2025b), i.e., the case where narrow finetuning leads to broad unintended behaviours, a natural question to ask is: Does fine-tuning recruit a behavioural substrate already present in the base model, does the adapter itself contain the mechanism, or does the behaviour depend on both?
The answer determines which defences can work. An adapter-localized mechanism can be audited and excised pre-deployment relatively cheaply. A base-resident one, however, means removal fundamentally trades against the capabilities that subnetwork also serves, and any defence against such behaviour must move upstream to pre-training.
Candidate settings: Emergent misalignment (Betley et al. 2025a; Turner et al. 2025), subliminal learning (Cloud et al. 2025), and inductive backdoors (Betley et al. 2025b)
Betley et al. 2025a: https://arxiv.org/abs/2502.17424 Betley et al. 2025b: https://arxiv.org/abs/2512.09742 Cloud et al. 2025: https://arxiv.org/abs/2507.14805 Frankle and Carbin, 2018: https://arxiv.org/abs/1803.03635 Renda et al. 2020: https://arxiv.org/abs/2003.02389 Turner et al. 2025: https://arxiv.org/abs/2506.11613
Theory of change
Narrow finetuning on innocuous-looking/unrelated data can install broad unintended behaviours. Where the mechanism lives determines which defence can work: an adapter-localized substrate can be audited and excised before deployment, while one already present in the pretrained model cannot be removed without trading against the capabilities it also serves. Existing mitigations assume some form of localization without establishing where it sits. This project measures that. A null result is equally informative, since it would constrain the localization assumption directly.
Your role
Mentees will have autonomy to propose and carry out their experiments as they see appropriate, as well as set the direction of various sections of the project. While I am happy to step in and suggest ways forward, troubleshoot, and brainstorm ideas, I would be very happy to support and guide experiments/ideas that mentees might have in the general realm of the project
Prerequisites
- Fluency with Python + PyTorch
- Familiarity with ML + Linear Algebra at the undergraduate level
- Comfort (at least conceptually) with fine-tuning and evaluating LLMs
Application question(s)
- Of the mentioned candidate behaviours (emergent misalignment, subliminal learning, inductive backdoors), which one would you start investigating? Pick one, argue for it, and propose the first experiment. Your answer should say what you would measure, what result would tell you the setting is workable, and what would tell you to switch. (300 words)
About the mentor
I'm a postdoctoral researcher at St. Jude Children's Research Hospital, working at the intersection of mechanistic interpretability and computational neuroscience. My PhD thesis studied structure–function relationships in biologically constrained neural network model organisms of cortical circuits, and I've since carried that structural focus into language models — most recently on the weight-space geometry of emergent misalignment, where we found shared low-dimensional parameter subspaces across independently misaligned models and used them to build weight-space interventions (https://darturi.github.io/shared-em-subspaces/).
In the past year I've worked with ~30 mentees in total, ranging from high schoolers to post-PhD researchers in the mech interp + AI safety space, producing eight accepted workshop papers at NeurIPS, ICLR, and AAAI. As a mentor, I do my best to be provide support as needed with brainstorming, experimental design, and analysis of results, while mentees lead their own workstreams, are encouraged to take ownership over their piece of the project, and pursue questions they find most interesting.