Work on a method for solving the Unrestricted Adversarial Examples Challenge (https://arxiv.org/abs/1809.08352).
About the project
You will join an existing team of 3 (include me) working to develop a method to solve the Unrestricted Adversarial Examples Challenge (UAEC) (https://arxiv.org/abs/1809.08352). This is a long-standing open problem in the field of adversarial robustness (see below for why we think adversarial robustness matters for safety).
We call the method we are working on Unrestricted Adversarial Training (UAT). We are trying to develop a version of UAT that works in the image classification setting and hope to apply to it to solve the Unrestricted Adversarial Examples Challenge.
UAT works similar to standard adversarial training, except instead of perturbing an image with a bounded perturbation and then training the model to predict the attacked image as the original class, we instead perturb an image with an unbounded gradient-based-perturbation towards a target class, and then train the model not the classify the attacked image as the target class. The model is also allowed to "abstain" from classifying any image.
Why do we think UAT will work? In our current experiments, we observe that as UAT training progresses, the images perturbed to look like class C look more and more like real class C images. As we scale up training, we believe that eventually the only attacks that can successfully fool the model into thinking an image is of class C will be genuine instances of class C. But if this occurs, then we have essentially solved the UAEC.
However, note that if you train using UAT for too long, you will eventually start training your model to classify realistic images of class C as being not of class C. To avoid this, one needs to stop training UAT from overshooting, and the exact place to stop depends on the part of the distribution you are working with. This is the core challenge we do not yet have a great solution to and are trying to fix.
Theory of change
Adversarial robustness is very closely related to the worst-case performance of a system. Safe systems are ones which by definition have acceptable worst-case performance, so adversarial methods can serve as both a good auditing mechanism and as a training signal for safety.
Additionally, our inability to steer advanced AI systems in a robust way reflects our inability to control the “core” values and tendencies of AI systems. For example, when RLHF is conducted on a LLM, it behaves aligned in the average case, but the existence of phenomena like “jailbreaks” shows that we have not effectively changed the “true” values of the system. Rather we have only instilled in it a bunch of heuristics that make it behave nice most of the time.
I think working on robustness is a good way to improve our ability to do alignment of “core values”.
It might also be that this goal is ill-formed – the intuition that generally intelligent systems have core values may be false and maybe the hot-mess theory of intelligence is right. Even in this pessimistic case, studying robustness could still give us highly decision-relevant knowledge about the nature of intelligence.
At the moment, I’m focusing on robustness in the vision domain as a stepping stone to robustness more broadly. My key focus is on developing techniques that can improve robustness against unrestricted adversaries.
I believe that if a technique like UAT is properly developed, it can also have meaningful applications to language models. In particular, I think UAT will enable more aggressive application of latent adversarial training. One of the key problems holding latent adversarial training back is that standard adversarial training with bounded adversaries does not make much sense in latent space.
Your role
You will help work on developing theoretically principled versions of UAT and also work on both small scale and large scale experiments that get it to work for real.
Mentees for this project will not have too much high-level autonomy, your goal will be to work with the existing team to get UAT to work. An ideal mentee is someone who deeply understands the proposal above and is excited to work on it.
Prerequisites
- Be able to deeply understand the proposal above.
- Either {strong ML theory/math skills and okay experimental skills} or {okay ML theory/math skills and strong experimental skills}.
Application question(s)
Optimize for clarity of response, not length.
- Why wouldn't just scaling up the perturbation bound in standard adversarial training work for solving the UAEC?
- How might we apply UAT / a solution to the UAEC to latent adversarial training in LLMs?
- What's the most impressive theory-ish thing you have done? (2 sentences max)
- What's the most impressive implementation-ish thing you have done? (2 sentences max)
About the mentor
Tony Wang is a PhD candidate at MIT working on adversarial robustness. Much of his previous work and thinking has been on adversarial robustness. He has thought about the phenomenon both in simplified toy settings as well as in the setting of superhuman game-playing agents.