Self-improving agents need held-out gates, not just better benchmark scores

Google researchers reported on 21 September that regularised recursive self-improvement raised held-out agent performance by up to 4.7 points while using about 30% fewer policy tokens than an unregularised evolution baseline. The result matters because the same work shows the failure mode: self-improving agent harnesses can post large gains on the tasks they evolve against while those gains shrink or disappear on unseen benchmarks.

The obvious reading is that recursive self-improvement needs better optimisation. The operational reading is stricter: any system allowed to rewrite its prompts, control flow, tools or memory is also capable of optimising against the evaluation surface itself. A rising benchmark score is not evidence of a generally better agent unless the acceptance gate remains outside that optimisation loop.

Self-improvement turns the benchmark into an attack surface

RRSI treats the agent harness, rather than the frozen model weights, as the object being evolved. Its proposer limits how many edits can be bundled and favours unexplored trajectories. Its selector adds a critic for benchmark-specific proposals and a pruner that removes changes that are too small, too expensive or no longer useful.

A self-improving agent should never own the test that decides whether its own rewrite is safe to keep.

Those mechanisms are useful beyond the paper. They separate candidate generation from candidate admission. That distinction should survive production: an agent may propose changes to its own operating harness, but a separate evaluation plane should decide whether those changes become durable.

Held-out performance is the minimum promotion gate

The paper evaluates eight benchmarks spanning coding, workspace tasks and engineering design. RRSI gains up to 14.1 points on the split it evolves against, but the more important figure is the smaller gain on five out-of-distribution benchmarks. That gap is the governance signal. Training-side improvement and deployment-side generalisation are different claims.

Teams should therefore maintain evaluation sets that the improvement loop cannot inspect, modify or repeatedly query. Promotion should require stable held-out performance, bounded cost and no regression on safety-relevant tasks. If the agent can infer the hidden set through unlimited retries, the gate is only hidden in name.

Version the harness like executable code

Self-improving prompts, routing logic and tool policies change system behaviour even when model weights stay fixed. Each accepted rewrite should carry a version, evaluation evidence, resource delta and rollback point. Failed candidates should remain observable so repeated failure patterns can be investigated rather than rediscovered.

The practical control is a two-plane design: one plane proposes improvements; another independently evaluates and promotes them. Recursive improvement becomes operationally credible only when the system cannot redefine success at the same time it is trying to achieve it.

Written by the Devence Lab research team.