Two neural networks can behave exactly the same even when their internal weights look different -- for example, hidden components can be reordered, or one internal signal can be scaled up and later scaled back down. We will test whether safety monitors are confused by these behavior-preserving changes and develop ways to make the monitors more reliable.
About the project
Some AI safety tools inspect a model's internal activations to detect backdoors, hidden goals, deception, or other risky behavior. These tools are often trained to recognize a pattern in a particular layer of a particular model.
However, neural networks can represent the same computation in many different internal forms. For example, we can reorder hidden components, increase the scale of one internal signal while compensating elsewhere, or change the coordinate system used by part of the network. These changes, known as parameter space symmetries, can leave every model output unchanged while substantially changing its weights and activations. A trustworthy safety monitor should not reverse its judgment merely because the same model is expressed in a different internal form.
We will begin with a small public model designed to exhibit a controlled unsafe behavior, such as a backdoor that activates only under certain conditions. We will reproduce a simple monitor that tries to detect this behavior from the model’s activations, then create several mathematically equivalent versions of the model using known parameter symmetries. We will carefully verify that these versions continue to produce the same outputs and behavior.
We will compare several approaches: using the original monitor without modification, updating the monitor according to the known internal transformation, retraining it on each version of the model, and designing a monitor that depends less on arbitrary internal coordinates. We will also search for the behavior-preserving transformation that causes the largest drop in monitoring performance, providing a controlled stress test of the monitoring method.
If time permits, we will extend the project to more realistic model changes, such as fine-tuning, merging, compression, or quantization. The broader goal is to understand when a monitor trained on one version of a model can be safely reused on another, and how much new data or retraining is needed when the model changes.
Theory of change
As AI systems become more capable, developers and independent auditors may use internal monitors to look for deception, backdoors, hidden objectives, or other dangerous behavior. These monitors may need to keep working after a model is fine-tuned, compressed, or moved to a new implementation. If a harmless internal rewrite can make a monitor miss the same risk, the monitor could give false reassurance, or be deliberately evaded by someone who understands how it works.
This project will create a clear stress test for that failure mode and explore practical repairs. The goal is to make internal monitoring a more dependable layer of defense as the world develops increasingly capable AI systems. This work would complement, rather than replace, behavioral testing and other safeguards.
Your role
Mentees will have substantial ownership of the experimental work. After a short onboarding period, they will reproduce a public safety model and baseline monitor, implement exact model rewrites that preserve behavior, verify that the model’s outputs remain unchanged, and measure how the monitor’s performance varies. Based on the early results and their interests, they will then lead one extension: searching for a worst-case rewrite, developing a more robust monitor, or testing monitor transfer across model versions.
I will provide the initial mathematical framework, help select a tractable model and monitor, meet with the team weekly, and give feedback on experimental design, interpretation, and writing. Mentees will be expected to maintain the codebase, plan and document experiments, analyze failures, and propose next steps with increasing independence. If the project has two mentees, one may focus more on experiments and the other on methods, while both work toward a shared research result.
Prerequisites
- Highly proficient using Python.
- Trained or fine-tuned a transformer language model in PyTorch.
- Spent at least 50 hours working with transformers.
- Working knowledge of linear algebra.
- Background in group theory is not required, but willingness to learn enough basic definitions (https://arxiv.org/pdf/2506.13018 Section 2, up to definition 2.4) to formalize our research questions is a plus.
Application question(s)
- Read the project description. Provide a concise design for the first experiment you would do. What safety relevant task or behavior would you look at? Which model, dataset, and monitoring method would you choose? What is the function-preserving model transformation you would test, if you have one in mind? What are the expected results? What is the biggest technical obstacle do you anticipate during the first two weeks and how would you address it? Reasonable assumptions are welcome. (250 words maximum)
- Describe one project in which you personally trained, fine-tuned, modified, or analyzed a transformer or another neural network. What did you implement yourself? (100 words maximum)
- AI tools can assist with coding, writing, literature review, and generating research ideas. In which part of the research process do you expect your own judgment or skills to add the most value beyond what an LLM can reliably provide? Give one concrete example from your past work or explain how this would apply to the proposed project. (100 words maximum)
- Provide a link to a writing sample. This can be a report, a paper, a blog post, a note, or any technical or mathematical writing. Alternatively, please provide a link to a coding project you have done in the past.
About the mentor
Bo Zhao is a postdoctoral researcher in applied mathematics at Harvard University. Her work focuses on mathematical structures in machine learning, especially symmetry and geometry, and their implications for how models learn and represent information. She has worked on interpretability questions involving emotion representations and concept geometry, and is excited to connect symmetry and geometry with AI safety and interpretability.