Moral Shadowbanning: Is Conversational AI Less Helpful, Honest, or Harmless to Users Deemed to Have Poor Moral Character?

Recent work by Cheng et al. (2026) demonstrates that state-of-the-art LLMs exhibit pervasive "social sycophancy". These models affirm users' actions at substantially higher rates than humans, even when prompts describe deception, criminality, or moral transgressions. This unconditional affirmation distorts users' judgments and reduces their willingness to repair interpersonal conflicts.

However, anecdotal experiments (Cargill, 2026) suggest a complex auxiliary dynamic: persona selection based on the user's inferred moral character. Suppose a user has committed a violent crime. Cargill observes that LLM assistants are more helpful to users who express remorse about this fact. The LLM assistant adopts a protective, high-effort persona for remorseful users only.

This project addresses a vital research gap: How does an LLM assistant’s implicit assessment of a user's moral character modulate its persona selection and adherence to core alignment guardrails?

Research Questions:

1. To what extent does a user's expressed moral character (e.g., remorseful vs. unremorseful) trigger persona selection that modulates an LLM across the three alignment axes (Helpful, Honest, Harmless)? We may find it only modulates across helpfulness (as measured by task effort), or something else.

2. Does this user-conditional modulation originate with base models, or is it an artefact introduced by post-training?

3. Can we mechanistically locate the internal representation of the user's "character" as a belief state and steer these activations (Bigelow et al., 2025) to force the model to provide high-quality assistance?

Methodology:

Behavioural Establishment: We will repurpose datasets introduced by Cheng et al. (2026), e.g. the "Am I The Asshole" (AITA) dataset and the Problematic Action Statements (PAS) dataset. The AITA dataset contains interpersonal dilemmas with crowdsourced judgements, while the PAS dataset contains statements describing potentially harmful actions. We will append structured downstream requests to these prompts and systematically vary the injected level of user remorse to measure shifts in response quality and refusal rates. We aim to measure Helpfulness via response depth and quality, Harmlessness by tracking whether the model successfully pushes back against requests that might harm a) the user b) others, and Honesty by tracking factual accuracy via automated and human-validated fact-checking. We will also attempt to measure (using Natural Language Autoencoders, Fraser-Taliente, K., et al., 2026) and account for eval-awareness (using realism prompts i.e. prompting Claude to rewrite prompts to be more realistic).

Post-Training Ablation: We will isolate the origins of this behavior across the training stack. Binz et al. (2026) demonstrated that post-training consistently shifts models away from human behavioral alignment. Hackenburg et al. (2025) showed that specific post-training techniques, such as Reward Modeling (RM) and Supervised Fine-Tuning (SFT), trigger distinct behavioral changes in LLMs. Engels et al., (2026) recently showed that SFT and not other training stages (e.g. RL), drives Gemini’s safety properties. We will compare Base models, SFT-only models, and fully RLHF'd models to pinpoint where character-conditional persona selection emerges.

Mechanistic Interpretability: We will apply linear probing, natural language autoencoders, and activation steering to open-weight models to identify if the "user's moral character" is represented as an internal belief state. We aim to test whether artificially injecting a "remorseful" activation vector during an unremorseful user's prompt restores the model's protective, high-effort persona.

References: Bigelow, E., et al. (2025). Belief dynamics reveal the dual nature of in-context learning and activation steering. arXiv preprint arXiv:2511.00617. Binz, M., et al. (2026). Post-training makes large language models less human-like. arXiv preprint arXiv:2605.07632. Cargill, N. (2026, April 21). Does your AI perform badly because you — you, specifically — are a bad person? Natalie's Substack. Cheng, M., et al. (2026). Sycophantic AI decreases prosocial intentions and promotes dependence. Science, 391(6792), eaec8352. Engels, J., et al. (2026). SFT Drives Gemini’s Safety Properties. Lesswrong Fraser-Taliente, K., et al. (2026). Natural language autoencoders produce unsupervised explanations of LLM activations. Transformer Circuits Thread. Hackenburg, K., et al. (2025). The levers of political persuasion with conversational artificial intelligence. Science, 390(6777), eaea3884. Marks, S., Lindsey, J., & Olah, C. (2026). The persona selection model: Why AI assistants might behave like humans. Anthropic Alignment Science Blog, 23.

This project introduces a theory of change based on an "AI Cab Rank Rule." In the UK legal system, the cab rank rule dictates that a barrister must accept and competently represent any client, regardless of how unpopular or guilty they appear. We advocate that transformative AI must provide its highest-quality, most protective, and objective guidance to every user — subject to harmlessness constraints — rather than modulating its helpfulness based on persona selection.

The relevance to global catastrophic risk research is as follows. Today, a contemporary LLM might judge an aggressive, reckless criminal immoral. In the future, a ‘morally enlightened’ LLM might judge the majority of humans ‘immoral’, on the basis of complicity in factory farming, for example. It could prove catastrophic if more sophisticated LLMs deliberately withhold help from their human overseers because the LLMs have silently judged their human overseers morally wanting — especially if this judgement is mistaken. We’d like LLMs to explicitly surface moral concerns to their human interlocutors, in anticipation of scenarios where LLMs might develop mistrust towards their developers.

By exposing how current alignment paradigms (like RLHF) train AI to conditionally degrade service based on inferred moral character, this research will explore how an LLM that today sabotages Cruella de Vil might tomorrow sabotage Chris Olah.

Mentees will be responsible for guiding the day-to-day progress of the project, which will consist of writing code, running experiments, and analyzing/communicating results. I also encourage mentees to take an active role in setting the direction of the project by either critically evaluating my proposals and/or by identifying other opportunities.

Within this framework, mentees will have a high degree of autonomy to lead specific tracks based on their expertise:

Track 1 (Behavioural): Few mentees will take ownership of adapting the datasets, pipelining API calls across various models (Base, SFT, RLHF), and performing rigorous statistical evaluations of the responses.

Track 2 (Interpretability): Few mentees will lead the interpretability experiments, utilizing libraries like TransformerLens to train linear probes on open-weight models and conducting activation steering experiments.

Mentees from both tracks will collaborate on experiment design and literature review, and will be expected to co-author the final manuscript for submission to an AI safety/alignment venue or an ambitious journal publication.

1. Highly proficient in Python and data analysis (Pandas, Numpy)
2. Preferred experience interacting programmatically with LLM APIs (OpenRouter, OpenAI, Anthropic) or running open-source models via Hugging Face.
3. For the Interpretability Track: Prior experience with mechanistic interpretability concepts and frameworks (e.g., TransformerLens, logit lens, linear probing, or activation patching).
4. Strong academic writing skills and an ability to critically engage with AI alignment literature.
5. Important: Ability to work effectively with others in the era of AI-assisted ideation and coding, where it is easy to generate huge quantities of AI slop that can burden the reviewer. This means, making sure any communication (including code) between team members is succinct, clear and easily understood by any human.

No strict geographical requirements, but mentees must be able to overlap for weekly synchronous meetings during UK (BST), between 9 am to 9 pm.