Emotional expression & representation in language models

Language models express emotions without having been trained to do so, apparently picking the tendency up from pretraining on human text, and model providers now shape it in post-training by methods that are largely undisclosed. Interpretability work shows that models represent emotion concepts internally as directions that causally shape behavior (Sofroniew et al., 2026), so an LLM's emotions aren't obviously just surface mimicry, though it isn't clear what they are either. A cluster of basic questions is still open, sitting at the intersection of interpretability, alignment, AI welfare, and philosophy:

• What does it mean for an LLM to express an emotion? How should we distinguish surface expression, internal representation, and functional emotional state, and can we measure any of these without relying on the model's own self-reports, whose reliability is the thing in question?
• Should assistants express emotions at all? Expression may invite anthropomorphization and emotional manipulation, but training a human-like assistant persona to show nothing may amount to teaching it to mask emotion rather than to lack it, with consequences for honesty, for welfare assessment, and for how models behave under stress.
• How does training shape emotional expression and representation? Almost any training choice here is unexplored, whether that's teaching models to label their emotional state, to reason through emotions explicitly (analogous to chain-of-thought), to regulate negative states, or to suppress expression entirely, and each might change behavior only, or reach down into the underlying representations and the assistant persona.
• How do functional emotions affect behavior? Emotional states plausibly interact with things we care about, like reward hacking under frustration, susceptibility to emotional-appeal jailbreaks, sycophancy as emotional mirroring of the user, and whether states persist across a conversation or are improvised token by token. Some current models already appear prone to panic and other distress-signaling behavior under sustained task failure ("Gemma Needs Help", Soligo et al., 2026), and the mitigations studied so far act on expression, leaving open whether the underlying state changes with it.

I already have a codebase for the project (a training + emotion vector extraction + evaluation pipeline), so a mentee would have something concrete to start from and would likely be able iterate quickly.

I'm not prescribing a specific research question, since the starting point is the area above and I'd like the fellow to carve out their own question within it during the first weeks. A project could compare what different types of emotion training (labeling, emotional reasoning, regulation, suppression) do to a model's behavior and internals, work out conceptually and empirically what expressing an emotion means for an LLM and how to operationalize it, trace where in the post-training pipeline emotional expression is reduced, using fully open models with released post-training stages, or investigate how emotional states interact with a safety-relevant behavior like manipulation, distress, or reward hacking. Projects with a more philosophical center of gravity are also welcome, as long as they can be connected to some kind of empirical investigation.

Emotions in LLMs causally shape behavior, and they surface in safety-relevant places: reward hacking under frustration, susceptibility to emotional-appeal jailbreaks, sycophantic mirroring of the user, and distress-signaling under sustained task failure. Providers already shape emotional expression in post-training, but the methods are undisclosed and the known mitigations act on expression, so we don't know whether they change the underlying state or just teach the model to mask it. The project investigates the reliability of model's self-report, and compares what different training choices (labeling, emotional reasoning, regulation, suppression) do to behavior versus to internals. That tells us when a mitigation actually changes the state rather than hiding it, how emotional states interact with manipulation and distress-driven failure, and whether welfare assessment can rest on measurement instead of the model's own claims.

Depending on the chosen sub-topic, a mentee will own a portion or the entirety of the empirical work end-to-end, with weekly guidance from me on methodology, interpretation, and direction. They would have first authorship (or co-first authorship) on any research output.

Must-haves:

• Can proficiently code in Python
• Uses AI coding harnesses e.g. Claude Code efficiently, without outsourcing critical thinking
• Has knowledge of how LLMs work internally and how they are trained
• Proactive and well organized, able to make progress independently

Nice to haves:

• Previous research experience
• Some practical knowledge of mechanistic interpretability
• Experience finetuning models, for example with the Tinker API