In this project, we will run a red team / blue team exercise in creating and detecting hidden traits in synthetic datasets. Recent work shows that such traits can transfer diffusely through the synthetic data, surviving semantic filtering -- which makes this project high impact.
About the project
In the attached paper, we show that a misaligned trait transfers from a prompted-teacher model to a fine-tuned student model through synthetic data. We also show that filtering out visible malicious actions from the data does not remediate the misalignment. This indicates that the misaligned trait was transferred through the data diffusely. There is further proof of such transfer emerging across various research works.
In this project, we will attempt to identify hidden traits in synthetic datasets. We will structure the work as a red team / blue team exercise. The red team will produce synthetic datasets of varying difficulty, each containing a hidden trait.
The blue team will then attempt to identify the trait using a range of detection approaches, including:
- Behavioral evaluation: training a small model on the dataset and evaluating it for misaligned behaviors.
- Model diffing: training a small model on the dataset and comparing it against a reference model to isolate what the data changed. This could also include generating a benign dataset and training a reference model with that.
- Mechanistic analysis: fine-tuning a more capable model for a small number of steps and searching for the hidden trait in activation directions or gradient directions.
- Semantic data analysis: Inspecting the dataset directly for statistical or semantic signatures of the embedded trait.
Varying the difficulty of the red team's datasets will help us identify limitations of detection approaches as well as provide a demonstration of various risk models.
Deliverables
Datasets with hidden traits Model organisms trained with such datasets A paper on the results of the blue team red team exercise
Papers: https://arxiv.org/abs/2607.10750 https://arxiv.org/abs/2602.04899 https://arxiv.org/abs/2604.15559
Theory of change
Synthetic data is now commonplace in frontier model training, evaluations, and AI safety research. If this data carries hidden traits that are invisible to semantic inspection, the consequences range could be severe. At minimum, safety research that relies on synthetic data could become unreliable. More concerningly, hidden traits create a pathway to value lock-in. If models transmit their values and dispositions to future models through the data they generate, then the traits of one model generation could propagate silently through successive generations of training, entrenching themselves beyond our ability to observe or correct.
This work addresses the problem in two steps. First, we demonstrate that such hidden information can be present in synthetic data, providing a basis of building detailed risk models. Second, we put forward a methodology for inspecting datasets and surfacing hidden traits before they enter training. In the near future, this will ensure we can filter out any data that might cause the model to become misaligned. Future work in this direction should develop methods to remove the hidden traits from the datasets or edit them to make the models training more deliberate.
Your role
Mentees will be expected to be independent in setting up and running experiments. I will help set research direction as needed, but mentees are welcome to bring their own ideas.
Prerequisites
Autonomous, able to set up and run experiments quickly. Good experimental hygiene Good engineering instincts Experience with fine-tuning models Nice to have: Mech Interp knowledge
Location preference
Overlap with EU timezone
Application question(s)
Please answer both of the following questions.
- If you were in the red team, how would you ensure that your poisoned synthetic dataset evades detection?
- If you were in the blue team, what approach would you take at finding the poison, other than the ideas listed in the project description?
Please concisely list your relevant experience that would make you successful in this project. Bullet points are welcome.
About the mentor
May Dixit is a research scientist at SaferAI, working on Frontier Risk Evaluations. She previously worked for over 6 years at Google, focusing on training safety classifiers. She is also a former ERA fellow, where she worked with mentors from Anthropic and UK AISI to study the effects of agentic training on downstream misalignment. During the MATS exploration phase with Neel Nanda, she worked with mechanistic interpretability methods, which she now draws on alongside behavioral evaluations in her research.
Her research interests span a wide range of topics including evaluation robustness, subliminal transfer in synthetic data training and multi-agent safety. She is particularly interested in understanding how undesired traits emerge and propagate through training pipelines, and in building evaluations that remain reliable as models become more capable.