We can use data attribution to find which documents will lead to the most (emergent) misalignment. Is there a pattern in the tokens that most contribute to it?
About the project
Data attribution aims to accurately estimate the impact that certain data points will have on the behavior of a model. We know that models can get generally misaligned when trained on narrow misaligned behavior (emergent misalignment), for instance when finetuned on bad career advice, models will generally display harmful attitudes towards the user. Data attribution techniques have been used to determine empirically which documents in the emergent misalignment setup lead to the most misalignment, having found that even though bad advice looks similar to us, some is disproportionately more impactful than others.
We will extend these results to token level attribution - looking for more precise interventions, as well as look for patterns that allow us to predict which of the tokens are the most influential. While several data attribution techniques have been found to be relatively good approximations of the actual influence of certain documents, sometimes beating influence functions, token level attribution is still under-explored.
Theory of change
that training a model on certain
Your role
I expect mentees to be doing all of day to day experiments as well as think about new experiments with my guidance.
Prerequisites
Have experience with Python and PyTorch (or know how to use LLMs and be able to evaluate how good the produced code is). Have done some work using model internals, or be ready to learn.
Location preference
None
Application question(s)
What do you think is the thing that is most likely wrong, or missing from https://drive.google.com/file/d/1BwUONlrJeTf1UmhLywXWFjM0I-xa3gtc/? (200 words)
What do you think is the most likelty reason for this project to not produce any relevant scientific knowledge (100 words)
If we have established that there are certain tokens that are significantly more influential than others, how would you go about classifying them/finding paterns that describe them (200 words)
About the mentor
I'm an interpretability researcher at EleutherAI. I studied physics in university (bachelors, masters and PhD) before realizing that making models safer and understanding them is more interesting and important. I worked on automated interpretability using sparse autoencoders and am currently interested in data attribution and how we can predict the effect that data will have on model's behaviors.