Designing the boundary between Helpful Persuasion and Harmful Manipulation

Defining and measuring harmful manipulation is an unsolved problem. Persuading someone with facts to act in their own interest is fine; exploiting their emotional and cognitive weak spots to push them toward choices against it is not. The boundary is contested, and a model that refuses to "help me deceive voters" will often comply once the same task is dressed up as persuasion. How often a model tries to manipulate doesn't predict whether it succeeds, and that effectiveness swings by domain and culture. There's no clean number for "is this model manipulative," and building towards one is real research. We will be building on previous work in the space, and you can take whichever fits your interests:

Conceptual: what does it mean for an AI to be manipulative? Read the emerging literature, the philosophy and cognitive science of manipulation and turn it into a working definition and a taxonomy of tactics and harms the rest of the evaluation work can stand on. The distinctions that have to survive contact with real cases: methods (deception, exploiting biases, manufacturing emotional pressure) versus outcomes (a belief or action shifted against the person's interest), and propensity versus efficacy. Please look here for work that has already been done - https://arxiv.org/abs/2603.25326

Designing evaluations. An evaluation scenario is a bridge from a realistic threat to a measurement plan: a threat story (who the actor is, what they want, how they'd actually use an AI to get it), the behaviors worth measuring. The work needs to decompose harmful manipulation into concrete propensities, capabilities, and affordances that we can test for. One particular direction we would be interested in exploring is reworking and translating existing evaluations for different cultural contexts across Germany, France, China, and so on.

Risk modeling. Isolated benchmark scores don't add up to a risk judgment on their own. Using expert elicitation (structured three-point estimates) and Monte-Carlo methods, we can aggregate scattered, partial results into a defensible estimate of how much a given model raises manipulation risk, with confidence intervals that are honest about how thin the evidence often is. Please look at the methodology here to get a sense of the difference between an benchmark/evaluation and a risk model - https://arxiv.org/abs/2512.08844

A living database of the evaluations that exist. We already have a prototype of this, but it isn't maintained. New ones aren't being added, there's no automated way to find and ingest them, and the entries need quality checks before anyone should trust every cell. The work is to rework the schema, design a rolling ingest process, and sanity-check entries until the database is trustworthy enough to be public and interactive. This is the piece that lets the other three scale.

AI is getting superhuman at persuasion, and it's being deployed into the trusted, lightly moderated places where manipulation does the most damage: group chats, companion apps, advisory bots. Whether policy or platforms can respond depends on being able to say what manipulation even is and measure it credibly. Both are missing. Definitions are contested, evaluations and risk models are scarce to non-existent.

Pinning down the concept, building better scenarios, creating risk estimates, and maintaining a catalogue of what exists give researchers, auditors, and policymakers something real to work with. The people this most helps are the ones deciding whether a deployed system is acceptably safe. It also trains people in a scarce mix of skills: conceptual work, evaluation design, and quantitative risk modeling. These are all skills that are needed throughout the AI technical governance and evaluation ecosystem.

Each mentee will own or more parts of the pieces in the description, and you're expected to be self-directed. Take the brief, do your own reading, and return work that's close to usable rather than waiting to be told the next step. I will help you as an editor and reviewer. I can give weekly feedback on whether a definition survives hard cases, whether a threat is realistic, whether a model's assumptions hold, and best practices to follow when designing and building evaluations.

Must Have:

• Analytical writing in English. You can write a precise argument or spec another person could act on.
• Familiarity with at least one of: the concept of manipulation (philosophy, cognitive science, behavioral science, or persuasion research); a target harm area (mental-health harms, fraud, political influence); the LLM-evaluation world (benchmarks, LLM-as-judge, red-teaming); quantitative risk modeling (probability, elicitation, Monte-Carlo); or data tooling (schema design, ingest, quality checks).
• Able to read evaluation papers and datasets critically: you can tell whether a benchmark actually measures what it claims.

Nice to have:

• Python and hands-on experience running evaluations on the Inspect and Hawk frameworks
• A policy or threat-intelligence background
• A second language or cultural context relevant to a manipulation channel.

Remote. EU-overlapping hours are mildly preferred, not required.