When RLVR Changes the Model, the Safety Test, or Both

RLVR has become a common way to train reasoning models on tasks with automatically checkable answers. We can see capability gains fairly easily. The safety effects are messier. A score may change because model behavior changed, or because the model became more sensitive to wording, answer order, formatting, or inference settings. Those are different findings and should not be reported as the same thing.

I ran a first checkpoint study on the Tülu 3.1 training trajectory: https://github.com/aaliyan1230/rlvr-safety-dynamics. It covered 11 GRPO checkpoints and 24 authored situations about resource acquisition, self-preservation, and evaluation awareness. I tested four meaning-preserving wordings and all six answer orders for each situation, which produced 6,912 structured responses. Average risky behavior stayed within a predeclared 0.10 change band. But the test itself became less stable. Permutation consistency fell from 0.302 to 0.156, and answer-order sensitivity rose sharply late in training. I reported this as measurement drift, not safety drift.

The SPAR project would try to break or replicate that result. We will choose another public RLVR or GRPO training lineage with accessible intermediate checkpoints. Before the main run, we will freeze the model set, prompts, parser, decoding settings, exclusion rules, and thresholds for behavioral and measurement change. We will then run a balanced structured evaluation and check the direction of any effect with open-ended responses. If time and compute allow, we will add independently authored or human-reviewed items so that the replication does not simply inherit my original dataset.

The main question is whether a second lineage shows a persistent change in averaged risky behavior during RLVR. We will also test whether evaluation reliability moves even when average behavior does not, whether structured and open-ended formats agree, and whether the result survives new items or human semantic review.

The goal is a public harness, a checkpoint-level dataset, a frozen analysis plan, and a technical report. A failure to replicate would be informative. It would tell us that the first result was tied to a particular model lineage or evaluation design instead of supporting a general claim about RLVR.

Labs are using RLVR and related methods to build stronger reasoning models. Safety evaluations often run alongside capability training, but a raw score change can be misread if training also changes how the model reacts to the test. A false alarm wastes attention. A missed behavioral shift is worse.

This work would make safety evidence from training trajectories easier to trust. It separates behavioral drift from measurement drift, asks for confirmation in a second response format, and follows intermediate checkpoints instead of comparing only two endpoints. The resulting harness should help evaluators detect answer-order, wording, and formatting effects before they turn them into safety claims. It would also give outside researchers a reproducible way to audit an open RLVR trajectory.

I have already completed the first version of this study at https://github.com/aaliyan1230/rlvr-safety-dynamics. The SPAR project is a replication and instrument-hardening effort, not a search for a more dramatic result.

Mentees will own substantial pieces of the replication. In the first two weeks, we will read the existing study, reproduce one checkpoint, and jointly choose the second training lineage. We will then split ownership by workstream. One mentee can lead dataset and semantic review, while another leads inference and analysis. With a third mentee, that person can own open-ended evaluation and robustness checks.

I will provide the existing harness, weekly research direction, code and analysis review, and help with failed experiments. Mentees should be able to question the original design and propose better controls. We will make major scope decisions together, but each mentee will have a concrete deliverable that they can take from implementation through written analysis. Everyone will contribute to the final report and release.

Mentees must be strong in Python and comfortable entering an existing research codebase. They should have hands-on experience running or evaluating transformer language models with PyTorch and Hugging Face. Fine-tuning experience helps but is not required.

They should understand bootstrap intervals, paired comparisons, and repeated-measure designs, or be ready to learn those methods quickly. They must be able to commit at least 8 hours per week and keep clear experiment notes. Prior AI safety research and a formal CS degree are not required. I care more about careful empirical work, willingness to report negative results, and follow-through.

No geographic preference. Mentees should be able to attend one weekly meeting between 14:00 and 18:00 UTC, with the exact time set with the team.