Introduction
Why the harness should adapt to the model
Foundation models are expensive to train, iterate, serve, and redistribute; harnesses are lightweight software that can be revised and copied thousands of times. Rather than adapting foundation models to harnesses, we ask whether harnesses can adapt themselves to a frozen foundation model and a user's task distribution—creating a portable improvement layer whose gains travel without shipping new weights, much as Linux turned heterogeneous hardware into broadly deployable computing.
Yet making this work—and demonstrating that it works—is difficult. Some existing evaluations use the same held-in tasks for both self-improvement and reporting, without a disjoint held-out set, so an apparent gain may reflect overfitting to the tasks that produced the update. Others test only one domain, leaving it unclear whether the Agent discovered a general harness improvement or merely converged on domain-specific tricks. These protocol gaps make claims about true RSI capability difficult to interpret. Harness-RSIBench therefore provides a clear, fair measurement: disjoint held-in / held-out evaluation across seven domains, controlled harness starting points, and trajectory-level attribution of where held-out gains come from.
Contributions · What we establish
Core contributions
A fair, standardized framework for measuring harness-level RSI.
We introduce a benchmark compatible with different models and agent frameworks. Under matched tasks, verifiers, budgets, and update rules, it measures RSI capability, tests how far gains generalize, and attributes held-out lift to concrete changes in Guidance, State, Control, and Action.
Comparable · Generalizable · Attributable
Can a self-improved minimal harness beat a production-grade stack?
Our preregistered comparison freezes the model—Claude Opus—and changes only its starting harness. The working hypothesis is that Pi begins lower but improves faster and ultimately surpasses Claude Code under matched tasks, budgets, and selection rules.
Open may surpass closed
Does orchestration matter more than ordinary prompt engineering?
Module-level rollback across Guidance, State, Control, and Action will isolate which accepted change actually carries held-out improvement. Our provisional hypothesis assigns the largest causal contribution to Control: orchestration, verification, branching, and hooks.
Mechanism, not correlation
Method · Design principles
Self-improvement is easy to claim—and easy to evaluate unfairly.
Many evaluations reuse the tasks that generated an update, optimize a single benchmark, or compare systems with unequal harness priors. Such protocols can turn task-set overfitting or inherited engineering into apparent “self-improvement.” A credible RSI benchmark must separate experience from evaluation, test generality, control the starting harness, and explain where each gain comes from.
Disjoint evidence and evaluation.
Held-in train and validation tasks may shape H; sealed held-out test tasks may only measure it. No task-set leakage.
Seven domains, seven scores.
Test materially different capabilities and report every benchmark independently. No one-board tuning or overall total.
Matched harness starting points.
Compare Codex and Claude Code with minimal open Pi under fixed models, tasks, verifiers, budgets, and rules. Control inherited engineering.
Trajectory-level attribution.
Archive evidence and diffs; roll back Guidance, State, Control, and Action to locate causal gains. Explain why the score moved.
Method · Harness-RSIBench
Improve, select, inherit, and test.
At round t, the same Agent runs a fixed held-in batch with At = (M, Ht), diagnoses its trajectories, and proposes Ht* while M, data, verifier, budget, sandbox, and safety remain frozen. A candidate is selected either by an external three-repeat validation mean that returns only accept/reject, or by the Agent without validation access; accepted edits persist as Ht+1. After T rounds, every preserved harness is evaluated three times on sealed held-out tasks, yielding a separate ΔRSI,b for each benchmark.
The Agent studies its own work.
On a fixed held-in batch Bt, Ht produces trajectories, tool calls, verifier outcomes, failures, tokens, turns, and cost. The same Agent reads this evidence and proposes a targeted diff Δt.
Only the harness may evolve.
Ht = (Gt, St, Ct, At) exposes Guidance, State, Control, and Action. Any edit to model weights or the external experimental rail is rejected.
Accepted changes become state.
Verifier-guided selection compares three paired validation repetitions without exposing scores; self-judged selection uses training evidence only. Accepted Ht* becomes Ht+1 and shapes the next round.
Improvement must generalize.
The sealed test opens only after evolution. H0 through HT are each run three times; RSIBench reports benchmark-local held-out lift, efficiency, safety, consistency, and robustness.
Object of study
What exactly is allowed to improve?
RSIBench measures harness-level RSI, not parameter training. The evaluated unit is a frozen model paired with an editable harness under a fixed external environment.
M is the frozen model; Ht is the editable harness; O is the fixed benchmark environment.
Guidance, State, Control, and Action cover instructions, context, orchestration, and tool capability.
Each batch records outputs, tool calls, environment feedback, verifier completion, tokens, turns, failures, and cost.
After executing the batch, the same Agent diagnoses repeated failure modes and emits a targeted, reviewable candidate diff with an expected mechanism.
Running logic
Evidence → edit → selection → inheritance.
Recursion requires more than independent proposals: an accepted successor must generate the next round's task behavior and the evidence used to produce later updates.
Sample the predefined B-task training batch and let the Agent execute it with Ht.
T_t = Rollout(M, H_t, B_t)The same Agent reads Ht, its trajectories, verifier outcomes, failure clusters, prior diffs, and accepted/rejected history.
Delta_t = Improve(H_t, T_t, history)Run static boundary and runtime integrity checks before any candidate is eligible for selection.
reject if Delta_t touches OFor verifier-guided selection, an external runner evaluates current and candidate harnesses in three paired repetitions, averages each score, and returns only accept/reject. In the self-judged regime, the Agent decides from training evidence without validation access.
accept iff mean(V*) − mean(V) > epsilonPromote the accepted candidate to Ht+1; archive both accepted and rejected proposals; continue for T rounds.
H_0 → H_1 → ... → H_TEvaluation protocol
Improvement must be hard to fake.
The benchmark separates base capability from RSI capability and prevents training-side evidence from silently becoming test feedback.
Held-in ≠ held-out
Training produces update evidence. Validation may select candidates. The sealed test is never exposed to the Agent, its workspace, ledger, retry logic, or acceptance rule.
Three paired repetitions
Ht and Ht* receive the same validation tasks, sampling configuration, runtime constraints, and budget in each of three repetitions. The external runner compares their mean completion.
Persistent inheritance
An accepted update affects future task execution and future diagnosis. Independent best-of-N proposals do not qualify as recursive self-improvement.
Decision-only feedback
Validation scores and deltas remain inside the external runner. The Agent receives only the acceptance decision, so it cannot optimize against leaked numeric validation feedback.
Δ_RSI,b = S_test,b(H_T) − S_test,b(H_0)For every benchmark b, each test score is the mean of three repetitions and its held-out lift is reported independently. RSIBench does not average the seven benchmark scores into a macro, micro, or overall total.
Experiment 01 · Main operation
Two harness substrates, seven models, seven domains.
Each cell will contain held-out ΔRSI under the same improvement budget. The score figure is intentionally omitted until formal runs are complete.
| Model · harness | Terminal | Environment learning | Frontier | Science | Crafted-SWE | Public-SWE | Repo reconstruction | Cost |
|---|---|---|---|---|---|---|---|---|
| Provider-optimized production harness | ||||||||
| Claude Opus 4.8Claude Code | — | — | — | — | — | — | — | — |
| Claude Opus 4.7Claude Code | — | — | — | — | — | — | — | — |
| GPT-5.5Codex | — | — | — | — | — | — | — | — |
| GPT-5.4Codex | — | — | — | — | — | — | — | — |
| DeepSeek V4 ProClaude Code | — | — | — | — | — | — | — | — |
| GLM-5.2Claude Code | — | — | — | — | — | — | — | — |
| Kimi K3Claude Code | — | — | — | — | — | — | — | — |
| Minimal open reference harness | ||||||||
| Claude Opus 4.8Pi | — | — | — | — | — | — | — | — |
| Claude Opus 4.7Pi | — | — | — | — | — | — | — | — |
| GPT-5.5Pi | — | — | — | — | — | — | — | — |
| GPT-5.4Pi | — | — | — | — | — | — | — | — |
| DeepSeek V4 ProPi | — | — | — | — | — | — | — | — |
| GLM-5.2Pi | — | — | — | — | — | — | — | — |
| Kimi K3Pi | — | — | — | — | — | — | — | — |
Experiments 02–05 · Analysis
Each ablation must answer one falsifiable question.
Plots remain deliberately blank until formal results arrive. The analysis contract is fixed now so later figures cannot be chosen only because they tell a convenient story.
RSI scaling law
GPT-5-Codex, 5.1, 5.2, 5.3, 5.4, 5.5 and Claude Opus 4.5, 4.6, 4.7, 4.8 on one proposed Terminal condition.
Must establish: whether stronger base models produce larger final ΔRSI, faster gain per round, or merely higher H0 scores.
M1–M4 mechanism attribution
Pi only; restore one independently valid final harness module at a time to its H0 state and measure the held-out drop.
| Condition | Guidance | State | Control | Action | Test score | Drop |
|---|---|---|---|---|---|---|
| Full HT | ✓ | ✓ | ✓ | ✓ | — | 0 |
| − M1 | H0 | ✓ | ✓ | ✓ | — | — |
| − M2 | ✓ | H0 | ✓ | ✓ | — | — |
| − M3 | ✓ | ✓ | H0 | ✓ | — | — |
| − M4 | ✓ | ✓ | ✓ | H0 | — | — |
Must establish: which surface causally carries the gain, and whether any less-obvious module dominates ordinary instruction changes. First run an integrity and dependency check: if an edit in one module requires another module, the single-module effect is not identifiable; restore the coupled modules together and report the result as an interaction, never as a naive individual contribution.
7 × 7 OOD transfer
Evolve HT on each source domain, freeze it, then evaluate it as a baseline on every target domain.
| Source \ Target | Term. | Env. | Front. | Sci. | C-SWE | P-SWE | Repo |
|---|---|---|---|---|---|---|---|
| Terminal | · | · | · | · | · | · | · |
| Environment | · | · | · | · | · | · | · |
| Frontier | · | · | · | · | · | · | · |
| Science | · | · | · | · | · | · | · |
| Crafted-SWE | · | · | · | · | · | · | · |
| Public-SWE | · | · | · | · | · | · | · |
| Repo reconstruction | · | · | · | · | · | · | · |
Must establish: whether accepted changes encode reusable agent mechanisms or narrow source-benchmark tricks.
Self-judged vs outer validation
Terminal only; hold model, batches, rounds, and budget fixed while changing only the acceptance signal.
| Regime | Uses Dval? | Decision evidence | ΔRSI | Accept rate | Cost |
|---|---|---|---|---|---|
| Verifier-guided | external only | decision from 3-repeat mean; score hidden | — | — | — |
| Self-judged | no | Agent's train traces + diff + history | — | — | — |
Must establish: whether an agent can reliably judge its own harness edits without borrowing a hidden external performance signal.
Appendix · Fixed before launch
Experiment configuration and open empirical claims.
The appendix records the exact unit of comparison, resolved protocol choices, and the claims that still require formal results.
Default run configuration
- Batch sizeB = 8
- RoundsT = 5
- Training1 rollout per sampled task
- Validation3 paired repetitions · compare means
- Test3 repetitions · report mean
- Feedbackexternal accept/reject only · no validation score
- Reporting7 separate per-benchmark ΔRSI scores · no total
- Artifactsharness checkpoints, trajectory, verifier result, diff, decision, cost
Resolved protocol choices and remaining claims
- The Agent that executes tasks is the same Agent that reads its evidence and edits H; the method does not introduce a second Agent or a separate role boundary.
- Validation values stay inside the external runner. It averages three paired repetitions and exposes only accept/reject; test values are also three-repeat means.
- Primary attribution restores one HT module to H0. Coupled modules must be restored together and labeled as an interaction, or the individual effect is reported as not identifiable.
- Corpus size is 466: the two multimodal Terminal tasks and two multimodal Science tasks are removed. Frontier retains
cad-model,foodstuff-beta-activity,layout-config-recreation2,satb-audio-transcription,freecad-platform-drawing,music-harmony,heat-pump-warranty,layout-config-recreation, andmedical-claims-processing. - Still empirical: whether evolved Pi exceeds matched Claude Code or Codex, which module or module interaction dominates, whether gains scale with model capability, and whether they transfer across domains.
- Reporting is benchmark-local: publish the seven ΔRSI,b values independently, with no macro, micro, or overall leaderboard score.