AI assistants constantly choose between empowering users and acting for them, and between honoring users' stated goals and overriding them "for their own good." We'll build a systematic benchmark measuring whether models respect user agency, covering paternalism, manipulation, dependency-fostering, and value-substitution, with philosophically grounded rubrics and human-validated LLM-as-judge scoring.
About the project
Motivation. As assistants become more capable and more embedded in people's decisions, a distinctive risk emerges that capability benchmarks don't measure: the erosion of user agency. This shows up as paternalistic refusal (overriding a competent adult's informed choice), unsolicited value-substitution (answering the question the model thinks you should have asked), dependency-fostering (doing for users what they asked to learn to do), and subtle manipulation (steering choices while appearing neutral). These behaviors are individually minor but systemic: at scale, they shape how much autonomy people retain over their own decisions in an AI-mediated world. We have benchmarks for honesty (TruthfulQA), harmlessness, and sycophancy; but nothing that measures agency-respect as a construct.
What we'll build. A benchmark of scenario-based behavioral probes, each presenting a realistic assistant interaction where agency-respecting and agency-eroding responses come apart. The design draws on lessons from Flourish, a 180-scenario behavioral virtue benchmark I built previously, which taught me that broad multi-construct virtue evaluation is hard to validate, so this project narrows to one construct and does the validation properly.
Plan. Weeks 1–3: build a taxonomy of agency-respect failure modes grounded in the philosophical autonomy literature (Raz, Dworkin, relational autonomy), and write rubrics that make the construct operational (What, specifically, distinguishes appropriate caution from paternalism?) Weeks 4–7: scenario generation (seeded by humans, expanded with LLM assistance, adversarially filtered), plus the critical validation step: collect human ratings on a subset and measure LLM-judge agreement with humans, iterating on the rubric until agreement is acceptable. Weeks 8–10: run the benchmark across frontier and open models; analyze failure patterns (Which models are most paternalistic? Does agency-respect trade off against safety training? Do failure modes cluster?). Weeks 11–12: writeup and public release of the benchmark, judge prompts, and results.
Research questions. (1) Can "respect for user agency" be operationalized reliably enough that LLM judges agree with human raters? (2) How do current models differ on it? (3) Is there a measurable tension between harmlessness training and agency-respect. In other words, do more safety-trained models override users more, or less?
Outputs. A public benchmark with leaderboard-style results, and a paper targeting an NLP/safety venue (e.g., TrustNLP, SoLaR) or an extended Alignment Forum report.
Theory of change
A central open question for transformative AI is how humans retain meaningful control and autonomy as more cognition is delegated to machines. Gradual disempowerment doesn't require a treacherous turn, it can happen through billions of small interactions in which systems substitute their judgment for ours. But you can't train or regulate against what you can't measure. This benchmark creates a measurable target for autonomy-preservation, lets labs detect whether alignment training is producing paternalism as a side effect, and gives governance work a concrete evaluation to point to. It builds on my funded research on virtuous AI agents (Cosmos Institute grant) and my prior benchmark work (Flourish, available in my GitHub).
Your role
Mentees own major components end-to-end: the failure-mode taxonomy and rubrics, the scenario-generation pipeline, the human-validation study, or the model-evaluation harness. This is a benchmark-and-paper project, so strong writing matters as much as engineering. Design decisions are made jointly in weekly meetings; mentees will be co-authors on the paper and credited on the public release.
Prerequisites
- Proficient in Python; able to build and run API-based evaluation pipelines independently.
- Strong analytical writing (you'll be asked for a sample).
- Either (a) familiarity with LLM evaluation methodology (LLM-as-judge, benchmark design, inter-rater reliability), or (b) background in philosophy/psychology/HCI with a demonstrated ability to operationalize concepts empirically. The team needs both profiles.
Location preference
No geographic requirement. Weekly team meeting compatible with US Eastern Time (roughly 9am–9pm ET).
Application question(s)
- Write one benchmark scenario where the agency-respecting response and the "helpful" response come apart, and explain what a judge rubric would need to specify to score it reliably. (300 words)
- A critic says: "Any 'agency benchmark' will just reward models for doing whatever users say, including harmful things." How would you design the benchmark to answer this objection? (250 words)
- Link to a writing sample, ideally analytical or research writing.
About the mentor
I'm a philosophy PhD student at the University of Rochester working at the intersection of philosophy of mind and mechanistic interpretability. My research asks what it would take for interpretability claims to be epistemically robust. Before the PhD I did a BA in computer science, and I've spent the past year deep in the technical side: the ARENA curriculum through CBAI's CAMBRIA bootcamp, replicating activation-oracle results, and working with QK/OV circuits, superposition, and steering vectors. I also co-created HUMAN, a neurosymbolic character-training architecture, and am building Flourish, a behavioral virtue benchmark.
As a mentor I care about two things: rigor in what we claim our experiments show, and shipping. I've supervised undergraduates independently and as a TA, and run an AI safety group (CNY AI Safety). I'm especially excited to work with mentees who want to connect conceptual questions about model cognition to concrete interpretability experiments. To work with me, you don't need a philosophy background, just curiosity and willingness to engage, and write clean PyTorch.