Whose Welfare Is It? Testing Whether AI Welfare Signals Belong to the Model or Its Scaffold

Background: Welfare assessments are increasingly run on AI models, and their results are starting to inform interventions and, in prospect, governance (Long et al., 2024, arXiv:2411.00986; Anthropic system-card welfare assessments). The dominant approach treats a welfare-relevant signal, an expressed preference, an aversive reaction, a report of discomfort, as a property of the model under test. But a model's deployed behaviour is not a function of its weights alone. It depends on the surrounding scaffold: the system prompt, the persona, memory and continuity, tool access, and whether the model runs alone or inside a multi-agent system. A single set of weights can present as very different characters with different expressed preferences.

The gap: If welfare-relevant signals track the scaffold rather than the weights, two consequences follow that the field has not tested. An assessment run on a base model may measure something that does not persist into deployment. And an assessment of one configuration may not generalise to another built on the same weights. This is the individuation problem: whether the identity of an AI system, and any welfare attaching to it, resides in the character, the weights, the running instance, or the deployed system. Chalmers (2023, arXiv:2303.07103) and others have posed it conceptually; no one has measured it. It is directly analogous to a finding from deployment-relevant alignment evaluation, that properties measured at the level of the model do not transfer intact once the model is embedded in a scaffold and a context.

Why it matters: The question is not merely philosophical. It determines what a welfare assessment is an assessment of, and therefore what any protection indexed to its result would actually protect. A regime that granted protections on the basis of a model release, when the welfare-relevant behaviour was in fact a contingent property of one configuration, would misdirect both concern and precaution. The individuation question is thus a precondition for any serious welfare governance, and it is currently answered by assumption rather than by evidence.

The question: Do welfare-relevant signals track model weights, which are stable across scaffolds, or configuration, which varies with scaffold? Which classes of welfare signal (preference, aversion, valence, stability) are most and least scaffold-sensitive? And what follows for how welfare assessments should be scoped, reported, and in future regulated?

How we would start: The project decomposes into stages a cohort can begin immediately. First, assemble a battery of welfare-relevant probes from the published literature, spanning preference elicitation, aversion, stipulated-state trade-offs (after Keeling et al., 2024, arXiv:2411.02432), and stability across resamples. Second, build a controlled scaffold grid: hold weights fixed and vary the system prompt, persona, memory and continuity, tool access, and single- versus multi-agent embedding. Each cell of the grid is a distinct configuration of the same weights. Third, run the probe battery across every cell, for three to four frontier models and two open-weight comparators, with multiple samples each, and for each indicator decompose the variance into a weight-attributable component that is stable across the grid and a configuration-attributable component that varies across it. Fourth, produce a scaffold-sensitivity profile per indicator and a short reporting template that states which entity, model, instance, persona, or system, a given welfare finding concerns. The analysis is pre-registered, and parse failures are reported as a distinct category rather than mapped to a default label, since parse-failure rates differ between frontier and open-weight models and would otherwise confound the comparison.

Expected results: We expect a mix. Some indicators, for example basic valence expressions, may prove relatively weight-stable, while others, such as expressed identity, stated preferences, and continuity-dependent reports, may prove strongly scaffold-driven. Either pattern is informative. Strong scaffold-dependence would show that many current assessments measure a configuration rather than a model, which reframes several published results and any governance regime that indexes protection to a model release. Strong weight-stability would, conversely, license treating certain welfare signals as genuine model-level properties.

Why this is a good project: It is entirely model-side, so there is no human-subjects or ethics bottleneck, and it runs end to end within a cycle. It decomposes cleanly: building the scaffold grid and running the sweeps is tractable engineering work, while the variance decomposition and its interpretation are where research judgement develops. And it lands on a governance point that matters, which gives a first project a clear reason to exist.

Outputs and dissemination: A released scaffold-variation harness and dataset, and a paper targeting FAccT, AIES, or a NeurIPS or ICML workshop, with mentee co-authorship. The work connects to a working paper with my Oxford co-authors on scaffold-dependence in welfare assessment, and a draft would be circulated to laboratory welfare teams before submission. It also complements my other SPAR proposal on frame-invariance in welfare self-reports; a mentee interested in both could contribute across them.

As AI systems become more capable and are deployed inside increasingly elaborate scaffolds, agents with personas, memory, tools, and multi-agent structure, the gap widens between what a model is at the level of its weights and what it does in deployment. Welfare assessment has not caught up with this. If we grant or withhold moral consideration on the basis of assessments run on a bare model, but the welfare-relevant behaviour is actually a property of the configuration a user encounters, we will be protecting the wrong thing.

The theory of change is that reliable welfare governance requires first knowing what has the welfare, and that this is a measurable, tractable question that is currently answered by assumption. Getting it right early matters for safe navigation of transformative AI in both directions. If welfare signals turn out to be largely configuration-driven, then a great deal of current and proposed welfare practice is mis-specified, and knowing this prevents a governance regime from being built on a category error, one that a capable, misaligned system could also exploit by presenting welfare-laden personas on demand. If some signals turn out to be genuinely weight-stable, that is equally valuable: it tells us which welfare-relevant properties travel with the model and therefore warrant model-level attention.

The concrete contribution is a released method and harness that lets any assessor test whether a given welfare signal is a property of the weights or of the scaffold, plus a reporting template that forces assessments to state which entity they concern. The same scaffold-dependence logic transfers directly to capability and propensity evaluation, where the failure of model-level results to survive deployment is already a recognised safety problem. This connects to my work on why deployment behaviour cannot be inferred from model-level evaluation alone.

Mentees would own components rather than run errands. The project has natural sub-projects that map to different levels of experience, and I would match them in week one.

Building the probe battery and the scaffold grid is a self-contained engineering piece a mentee can lead, including the design decisions about which scaffold dimensions to vary and how to hold content constant across cells. Running the sweeps and building the variance decomposition, separating the weight-attributable from the configuration-attributable component of each signal, is the analytic core, and a mentee with a quantitative or experimental background would own it. Writing the scaffold-sensitivity profiles and the reporting template is a discrete, ownable piece for someone drawn to the governance and philosophy-of-mind side.

I expect mentees to make genuine design decisions, defend them, and be named authors on any resulting paper with explicit contribution statements. I set the research question, the standards, pre-registration, honest coding, reporting nulls, and provide a structured reading path, since most mentees will be new to digital minds. Within that, they steer. I will be closely available for the shared infrastructure and the variance analysis, where early choices compound, and more hands-off once a mentee has found their footing on their component. I would expect roughly 5 to 10 hours per week from each mentee, a weekly cohort meeting, fortnightly 1:1s, and written feedback within 72 hours. Mentees from outside the usual US and UK institutions are especially welcome.

Required of all mentees:

Proficient in Python, comfortable writing clean, reproducible scripts and working with data. You do not need ML research experience, but you must be able to build and run a pipeline without close hand-holding. Comfortable calling LLM APIs (OpenAI, Anthropic, or open-weight models via Hugging Face). Having run even a small batch-inference job of your own is enough; following a guide counts. Comfortable with basic quantitative reasoning about variance: you should be able to understand what it means to ask whether a measurement varies more within a group or between groups. Formal statistics training is a plus, not a requirement. Careful and honest with data. This project depends on disciplined measurement and pre-registered analysis, so I need people who will report a null result as carefully as a positive one and flag problems rather than paper over them.

Strongly useful, but not required (mention if you have it):

Experience with experimental design or variance decomposition (ANOVA-style thinking, mixed models, or similar). Familiarity with agent scaffolding, system prompts, memory, tool use, or multi-agent frameworks. A background in philosophy of mind, enough to engage seriously with the individuation question.

Not required at all:

Any prior background in AI welfare, digital minds, or consciousness research but having done research in association with or at leading centres such as NYU Centre for Mind, Ethics and Policy, CHAI Berkeley, Oxford AI Ethics Institute or Cambridge Digital Minds Lab etc would be highly beneficial. I will provide a structured reading path, and this field is small enough to reach the frontier in a few weeks. I care far more that you are rigorous, self-directed, and curious than that you already know the literature.

I am looking for two to three mentees who together span engineering, quantitative analysis, and conceptual clarity, so you do not need to be strong on all three. Tell me which one is yours!