In this project, we want to understand how models process conflicting beliefs/preferences they learned during post-training or model spec midtraining.
About the project
As AI models become capable, understanding how their values are shaped over the course of different stages of training has become ever more important. In this project, we want to understand how models process conflicting beliefs/preferences they learned during post-training or model spec midtraining. To do so, we create synthetic documents similar to [1], and perform mid-training on Qwen3.5-4B base model on a mixture of different documents describing conflicting preferences (e.g. Honesty vs Task Completion, Corrigibility vs Autonomy). We then let the model go through the full safety fine-tuning and RL. We want to answer the following questions, including but not limited to:
(a) How does the mixture ratio between different preferences in mid-training documents affect the model's preferences after post-training? (b) How does the preference level injected into the model during mid-training change over the course of SFT/RL? How does the post-training methodology (e.g., SFT vs. RL vs. on-policy distillation) affect this change? (c) Qualitatively, how does the model reconcile conflicting beliefs? Do certain interventions make the model eventually pick one side, despite the training documents describing conflicting beliefs or behaviors?
[1] Measuring Reward-Seeking by Instilling Contrastive Beliefs, https://alignment.openai.com/measuring-reward-seeking/
Theory of change
As AI models become capable, understanding how their values are shaped over the course of different stages of training has become ever more important. This project's outcomes can help us learn more about how the current post-training pipeline affects the final model's values, and how to make the final model safer and more reliable.
Your role
Mentees will design experiments that can help us answer the questions proposed in the project description, run them, and organize the results into the form of a paper.
Prerequisites
- Highly proficient using Python
- Trained or fine-tuned a transformer language model in PyTorch
- Preferrably have some exposure to training/inference frameworks for Reinforcement Learning and Supervised Fine-tuning such as vllm, verl, SkyRL
Location preference
USA
About the mentor
I'm a rising fourth-year PhD student at MIT, broadly interested in AI safety, alignment, and scalable oversight. As a mentor, I'm looking for mentees who are curious, critical, and willing to keep asking questions, particularly questions that push back on our own implicit assumptions! I'm excited about understanding how modern LLMs' behaviors and values evolve over training, and how we can translate those learnings into making LLMs more reliable and safer.