69.4% became 45.9% when the server actually ran. Coding-agent gates need live serving.

NVIDIA's SWE-Serve benchmark tested 627 coding-agent patches on 19 inference-engineering tasks with live-serving checks. The same patches passed 69.4% of the time when those checks were removed, but only 45.9% with the complete verifier. That is 147 patches that looked acceptable until a real server had to load a model and serve requests.

The result is a deployment warning, not another model leaderboard. Coding agents can satisfy repository-level checks while breaking the runtime path that matters to users. For inference software, the release boundary has to extend beyond tests that never start the service.

Repository correctness misses runtime coupling

SWE-Serve contains 53 tasks derived from 83 merged SGLang pull requests. Nineteen tasks start a live server and exercise behaviour such as model loading, expert routing, OpenAI-compatible requests, batched generation and log probabilities. Three tasks also enforce calibrated performance gates on an H100.

That scope exposes failures that static checks and narrower tests cannot see. In one Gemma 4 MoE task, 16 of 33 patches passed every other check but failed at least one live-serving test. The code changed correctly enough to satisfy local expectations and still failed the product path.

An agent patch is not production evidence until the system it changed can load, serve and behave correctly through its real interface.

Cross-domain changes deserve a stronger gate

NVIDIA divided the serving path into request I/O, scheduling and lifecycle, model execution, and KV-cache or runtime-resource management. Tasks confined to one runtime domain passed at 69.0% across the best settings of 11 models. Tasks crossing multiple domains passed at 47.7%, a 21.3 percentage-point gap, and every tested model showed the same direction.

That gives platform teams a useful risk signal before a patch reaches production. A change touching scheduling, execution and cache state should not inherit the same verification depth as a local configuration edit. Agent-generated diffs need gates proportional to the runtime domains they cross.

Make live serving part of agent acceptance

A practical pipeline should separate code-generation success from deployment acceptance. Let the agent run unit and regression tests, then start the actual serving stack on representative hardware, load the target model and exercise the public API. Verify output ordering, log probabilities, batching, multimodal paths and performance where the change can affect them.

Keep the distinction visible in CI. A repository test pass can advance a patch to runtime verification; it should not label the patch deployable. Record which runtime domains changed and require the corresponding end-to-end tests before merge or rollout.

NVIDIA is careful not to claim that a SWE-Serve pass makes a patch merge-ready or endorsed by SGLang maintainers. Production teams should keep that restraint. The benchmark's useful lesson is narrower and actionable: when coding agents modify inference infrastructure, live serving is not an optional confidence boost. It is part of the correctness definition.

Written by the Devence Lab research team.