Optimizing LLMs with small, interpretable updates

In our experiments, we’ll start with a trusted model T, then give it the minimum amount of information necessary to improve its performance. That information might come from a smarter, untrusted model U which might want to cause T to take misaligned, catastrophic actions. (The trusted/untrusted model taxonomy originally comes from AI control: https://www.lesswrong.com/posts/kcKrE9mzEHrdqtDpE/the-case-for-ensuring-that-powerful-ais-are-controlled.)

It can be productive to think about minimizing the amount of information that T receives from U as it performs the task. That is, we can put an information bottleneck on U’s outputs. I claim that information bottlenecks improve safety for these reasons:

1. U’s outputs may be too restricted to steer T towards a malicious action that it would otherwise be highly unlikely to take.
2. In most experiments, U must communicate to T in natural language that is legible enough for T to act on, so we can expect a trusted LLM monitor to understand the advice too.
3. Even when a malicious action is understandable in principle, it’s faster and easier for a human auditor to read tiny amounts of information than long, complicated actions.

Even when U cannot cause T to take catastrophic actions due to a narrow information bottleneck, we hypothesize that useful actions are relatively easy for U to induce in T. While T would be very unlikely to cause a catastrophe on its own, it is already predisposed to try to complete the task, and many useful actions that U might suggest are easy for T to understand.

I’ve done two major projects involving the concept of information bottlenecks (see “Past research” below). There are many other questions in this area that I’m interested in answering. To name just some of them:

• Untrusted advice (see blog post #1 in “Past research”):
• How can we improve U’s advice?
  • I suspect that with better elicitation (e.g. doing RL on U), we can improve usefulness while keeping the amount of advice fixed.
• How can we red-team the untrusted advice protocol?
  • That is, if we widen the information bottleneck by letting U write longer advice, can U cause T to execute a malicious side task in the BashArena setting? How can we counter the red team by putting different kinds of restrictions on the advice?
• Prompt/scaffold optimization (see blog post #2 in “Past research”):
• Can we find a system prompt that replicates the behavioral change induced by RL training?
  • This could provide a way to do “model diffing”: by reading the system prompt, we get an idea of what changes training has caused, including changes we may not have intended.
  • See the questions below for a simple idea for an experiment of this kind.
• Can a trained model introspect about the changes induced by its training and write a prompt or scaffold for the untrained model that accurately captures those changes?
• Interpolating between U and T:
• We can interpolate the token distributions of U and T in a way that limits how much optimization power U can exert. Given this, is it the case that U can improve T’s performance but is unable to cause it to take malicious actions?
  • See the appendix in my post on untrusted advice for more about how to do this interpolation.
  • You can think of this as U choosing between a large set of rollouts that T thinks are sufficiently probable. This effectively makes U into a “satisficer” rather than a typical utility maximizer: https://www.lesswrong.com/w/satisficer
• Training with a tiny dataset:
• The literature on dataset distillation is about making the smallest dataset that teaches a skill. Can we replicate the effects of RL using a tiny inspectable dataset that is much smaller than the sum of all rollouts during RL?
  • These datasets are not always interpretable, and it might not always be clear how exactly they affect the model’s behavior. We should start by searching through the literature to understand various techniques and how they balance interpretability with making the dataset small.
• Training a small number of weights:
• If we fine-tune T with only ~10 trainable weights, does this limit the change in T’s behavior to be “simple”?
  • If so, maybe we can improve T’s performance, but we can’t get it to secretly take catastrophic actions.
• Combining multiple tiny weight diffs in a scaffold
  • To improve expressivity, we could make multiple minimal changes to the weights and turn them on or off based on minimal classifiers, similar to Base Models Know How to Reason, Thinking Models Learn When.
• In these experiments, there may not be an untrusted model U, but the training process itself can cause misalignment and we can think about bottlenecking information from it.
• Other experiments that aren’t on this list, including your own ideas!

Past research:

I wrote a blog post that explains the concept of information bottlenecks and how they could be useful for AI control. I recommend reading this to understand the kind of research we’d be doing, including “Appendix: more ways to implement information bottlenecks”, which contains more detailed descriptions of some experiments mentioned above. The post is called “Untrusted advice for AI control: Short, strong advice significantly uplifts weak LLMs” and it should be on my LessWrong profile soon (https://www.lesswrong.com/users/caleb-biddulph). If it’s not there, see https://docs.google.com/document/d/1QrRmkn07\_er0zTEWxHkNEibtbyz8GTYVW3kGF72vKZw.

If you have time, I also recommend taking a look at the blog post about my paper “Prompt Optimization Makes Misalignment Legible” (https://www.lesswrong.com/posts/vRpLPZpmECCfxHfv6/paper-prompt-optimization-makes-misalignment-legible), which is another example of research that involves information bottlenecks.

This research could advance the AI safety subfields of black-box interpretability and AI control. Like other research in these areas, this could help us understand when AI tries to do something harmful and stop it, or simply remove the affordance for it to do something unusual enough to be harmful.

See the explanation above, as well as my “Untrusted advice for AI control” post.

Mentees should be able to do the coding work pretty much on their own, but they can feel free to send me questions over Slack, and I can meet with them weekly to discuss research direction. Mentees may work in groups of 1-3, depending on interests.

If a mentee just executes on what I suggest they do each week, this would be acceptable. However, I would be excited for mentees to take the initiative to suggest their own experiment ideas or even develop their own related projects, as long as they talk to me about it and I think it's a good idea. I think this is good for their personal development as researchers.

• Highly proficient in Python.
• Has experience with an AI coding assistant or able to pick it up.
• Comfortable with concepts in ML, LLMs, and AI safety (train-test split, reinforcement learning, prompt caching, token logprobs, reward hacking, etc.).
• Interested in gaining research taste and becoming self-directed, learning to identify the next interesting question to ask.
• Experience with training LLMs is useful but not required; some subprojects may only involve LLM inference.

I live in Berkeley, California. If you happen to be in the area, it might be nice to have an in-person meeting once in a while, but remote work is totally acceptable. Ideally, you should be available for a 30-60 minute meeting once a week between 9 AM and 5 PM PST.