Influence functions use gradients of model parameters to find the data points that most affect a model behavior. Can we use them to locate where certain knowledge is located in models, and use that for more targeted unlearning?
About the project
Current unlearning tools are very blunt (and don't really work). LACUNA is a benchmark with models trained to have localized knowledge - only certain groups get parameter updates when trained on that knowledge - and so can be used to measure how blunt a certain unlearning/knowledge location technique is. During the process of computing the influence of a certain data point, influence functions compute gradients for all parameters of the model. One could look for knowledge localization signals by finding which (sets of) parameters are the most influential for a certain behavior, in this case disclosing personally identifiable information.
Theory of change
Data attribution is significantly under-explored technique in the AI safety community. Being able to better understand how training changes models parameters might help develop better unlearning techniques. Unlearning has clear application in alignment, either by using it to remove harmful/dangerous knowledge a model might have learned or by using it to update out of data information
Your role
I expect mentees to be able to code and run their own experiments. I will be working with mentees directly to design said experiments but would expect them to be able to come up with interesting experiments by themselves.
Prerequisites
Proficient at Python and Pytorch (or be able to prompt LLMs and evaluate the legitimacy of their outputs) Have done experiments intervening on model's internals (same LLM clause as above)
Application question(s)
Why is robust unlearning important and why do most most unlearning methods fail at that? (200 words)
Why would knowledge localization help with unlearning? (200 words)
What would be your first experiment to localize knowledge given a forget set, a train set, model completions we would want to avoid, and model completions we would want to allow (400 words)
About the mentor
I'm an interpretability researcher at EleutherAI. I studied physics in university (bachelors, masters and PhD) before realizing that making models safer and understanding them is more interesting and important. I worked on automated interpretability using sparse autoencoders and am currently interested in data attribution and how we can predict the effect that data will have on model's behaviors.