Constitutions and Reasons: virtue-based character training with reflect-update correction loops

Background. Character training is now a load-bearing part of frontier post-training, and two recent results define the state of the art. Maiya et al.'s Open Character Training (arXiv:2511.01689) is the first open implementation: hand-written constitutions of first-person trait assertions drive DPO distillation from a teacher model, followed by SFT on synthetic introspective data, with code, checkpoints, and a strong evaluation suite released. Anthropic's “Teaching Claude Why” (Kutasov et al., 2026) describes what worked in production: responses that reason from principles matched behavior demonstrations 28x their size (with response rewriting for alignment quality the single most critical step, a 19x effect), and third-person disposition documents (“Claude does X, prefers Y, weighs Z over W”) trained as pretraining-style documents outperformed the same content in chat format. In short, demonstrations teach the what; the gains came from adding the why and the who. OCT's pipeline is almost entirely the what: teacher demonstrations without articulated reasons, trained with DPO, a method Anthropic's production pipeline notably does not use for character and which OCT itself had to stabilize with added KL and NLL terms. The approach. Virtue ethics offers a specific account of how character is acquired: through habituation across varied situations, reasoning about why actions fit or fail dispositions, and ongoing correction against consequences, rather than through imitation of demonstrated behavior. This project operationalizes that account as three components. (1) Excess/mean/deficiency contrastive data. For each scenario, we generate three candidate responses: one exhibiting excess of the target trait, one the appropriate mean, one deficiency. Training teaches the mean as the target, with both failure directions as contrasts. This is strictly richer than chosen/rejected pairs, which only contrast in-character against out-of-character: it teaches that a trait is a calibration, not a direction to maximize. Many documented character failures are excess failures (sycophancy as excess agreeableness, paternalism as excess caution), and demonstration data cannot teach the difference between a trait and too much of it. (2) Rationale-annotated responses. Target responses argue through the considerations at stake, including weighted disagreements where legitimate values conflict, supplying the why. Rationale annotation uses a rewriting step analogous to the one Anthropic identified as their largest effect. (3) A reflect-update correction loop. Rather than freezing character after one training run, we iterate: the trained model acts in scenarios (including model-to-model interactions, extending OCT's own self-interaction machinery), consequences are scored by an evaluator against explicit disposition criteria with human spot-checks for judge validation, and reflection data from scored outcomes drives the next round of fine-tuning. We run two to three rounds and measure whether character improves across rounds and whether the loop outperforms the same compute spent on a single static run. No current character-training work implements iterated correction. Throughout, the virtue taxonomies, triple structure, rationale formats, and scoring criteria are explicit and human-readable, so the character installed in the weights is traceable to legible specifications rather than to an opaque preference signal. This builds on my prior work on virtue-based training architectures (with Danny Potocki, supported by a Cosmos Institute grant). Research questions. (1) Can this pipeline be built openly on OCT's infrastructure, with quality-filtered SFT in place of DPO? (2) On matched traits, starting from OCT's flourishing persona as the natural bridge case, does it beat the OCT baseline on OCT's own metrics (revealed-preference coherence, robustness to adversarial prompting and prefill attacks) and on held-out moral generalization, meaning novel scenarios neither method saw in training? (3) Does the correction loop produce measurable improvement across rounds? (4) Which ingredient does the work? Ablations isolate DPO versus SFT, chosen/rejected pairs versus excess/mean/deficiency triples, demonstrations versus rationale-annotated responses, and one-shot training versus the loop. Plan. Weeks 1-2: reproduce OCT from its released code and checkpoints for one or two personas; establish baselines on its eval suite. Weeks 3-6: build the data pipeline (scenario generation, excess/mean/deficiency triples, rationale annotation, evaluator criteria). Weeks 7-9: train matched models, run the head-to-head evaluation, ablations, and correction loop. Weeks 10-12: optional interpretability comparison using persona vectors (Chen et al., arXiv:2507.21509), writeup, open-source release. Risk and fallback. Scoped to degrade gracefully: if training iteration proves too heavy for twelve weeks, the data-level comparison (chosen/rejected versus excess/mean/deficiency versus rationale-annotated data, evaluated directly with LLM judges) plus a single loop iteration are independently publishable. The OCT release removes the cold-start risk that usually kills training projects on this timeline, and Anthropic's published ablations give strong priors on experiment priority. Outputs. An open-source virtue-based character-training pipeline extending OpenCharacterTraining; an empirical comparison with ablations; the first test of iterated correction-loop character training. Alignment Forum post at minimum, workshop paper if results warrant. Since Anthropic's methods are described but not released, an open implementation of principle-grounded character training is a contribution independent of how the virtue-based approach fares.

Character training is how labs actually shape model values, and with the OpenCharacterTraining release, the open-source ecosystem is about to adopt demonstrations-plus-constitutions as its default paradigm. Anthropic's production experience already suggests that paradigm produces brittle alignment that fails out of distribution, and that the fix is training on reasons and identity, not just behavior. Testing this openly, with ablations, before the paradigm hardens is the point of this project. The excess/mean/deficiency framing additionally targets a failure class current methods handle poorly: excess failures such as sycophancy and paternalism, where the problem is not a missing trait but a miscalibrated one. If virtue-based training with correction loops produces characters more robust to persona jailbreaks and drift, that is a directly deployable improvement to open post-training and an open implementation of methods Anthropic has described but not released. If it does not, a careful negative result disciplines a growing body of work on virtue ethics for AI, including my own, with empirical evidence. A further benefit of legible training scaffolding is auditability: when character comes from explicit specifications and structured rationales, character regressions can be traced to specific training inputs, which opaque preference pipelines do not allow. Related prior work: my Cosmos Institute-funded research on virtuous AI agents, and my paper on the detecting and steering LLMs empathy-in-action (arXiv:2511.16699).

Mentees are hands-on builders. The three workstreams are: reproducing and extending the OpenCharacterTraining baseline, the virtue data pipeline (scenarios, excess/mean/deficiency triples, rationale annotation), and the evaluation, ablation, and correction-loop harness. Because this pipeline exists on paper but not in code, mentees make genuine design decisions rather than executing mine; we resolve design questions together in weekly meetings. Co-authorship on all outputs, with order reflecting contribution.

• Highly proficient in Python and PyTorch.
• Has fine-tuned a language model at least once (LoRA, SFT, or DPO on an open model; toy scale and following guides is fine).
• Comfortable in the huggingface ecosystem (transformers, peft, trl or equivalent) and able to run someone else's released training code.
• For at least one team member: serious interest in ethics, enough to engage with what a virtue is and how to operationalize one. No philosophy credentials required.

No geographic requirement. Must be available for a weekly team meeting compatible with US Eastern Time (roughly 9am to 9pm ET).