Synthetic agent data needs executable verifiers, not plausible prompts

ServiceNow CoreAI’s AutoSynthData generated 2,000 synthetic enterprise-agent training samples in about 18 hours in one EnterpriseOps Gym experiment. Fine-tuning on that set raised mean Pass@1 by 7.2 percentage points, a 35% relative gain. The useful result is not the volume. It is how aggressively the pipeline tests whether each generated task deserves to become training data.

AutoSynthData starts from capability gaps: cases where a target model fails and a stronger teacher succeeds. It converts those gaps into capability specifications, then generates fresh tasks with different entities, environment states and solution paths. That avoids simply replaying evaluation prompts and turns failure analysis into a moving curriculum.

A verifier is part of the training example

ServiceNow defines an agentic task as a system specification, a user prompt and a verifier. That third component changes the data-quality problem. A realistic-looking request is useless if the training pipeline cannot reliably determine whether the resulting environment state is correct.

AutoSynthData therefore applies a positive gate to confirm that the intended solution passes, then a negative gate to check that mutated incorrect outcomes fail. Candidates can enter a bounded critique-and-repair loop before rejection. For enterprise teams, this is the control to copy: generated tasks should not enter supervised fine-tuning merely because another model says they look plausible.

Synthetic agent data is only as trustworthy as the executable test that decides whether the agent succeeded.

Generate from failures without reproducing them

The generator works from sanitised capability cards rather than the original evaluation prompts, entities, trajectories or verifier details. New samples must exercise the missing capability while changing the surrounding state and solution path. A batch-level review then checks coverage, diversity and redundancy so one easy task family does not dominate the dataset.

That separation matters for internal agent programmes. Evaluation sets often contain sensitive workflows or must remain uncontaminated for later measurement. Teams can preserve that boundary by exporting only capability descriptions into generation, keeping original test cases and verifier internals outside the generator’s context.

Measure whether the curriculum closes the actual gap

The second EnterpriseOps Gym run shows why improvement must be measured against the original weakness. In the ITSM domain, ServiceNow reports 1,994 generated samples and mean Pass@1 rising from 18.77% to 27.18%. Those figures belong to the tested models and environment, not to enterprise agents generally.

The operating pattern is portable: record target-model failures, identify capabilities where a stronger reference succeeds, generate new environment-grounded tasks, execute both positive and negative verifier tests, then retrain and re-evaluate on held-out work. Synthetic data becomes useful when every sample has a falsifiable success condition and the next curriculum is driven by what the deployed model still cannot do.

Written by the Devence Lab research team.