A fluent agent can still choose the wrong skill. Evaluate routing and execution separately.

AWS added skill-focused evaluation for agents on 22 September, separating whether an agent chose the right skill from whether it actually followed that skill's instructions. That distinction matters because both failures can still end in a fluent, plausible answer.

A single end-to-end quality score hides the diagnosis. If an agent selects the wrong procedure, rewriting the procedure will not fix routing. If it selects the correct procedure but skips a mandatory step, improving the router will not fix execution.

Routing and execution need different gates

AWS's Skill Selection Accuracy evaluator returns a binary result for each invoked skill. Skill Instruction Following uses five levels, from Fully Followed to Not Followed, and passes at Mostly Followed or better in the documented example. Strands Evals also provides Skill Invoked, a deterministic check that a named skill loaded.

That creates a useful release pattern. For workflows where the correct procedure is known, assert the required skill deterministically. Then score whether the chosen skill was appropriate and whether its prescribed steps were completed. Keep the three results separate instead of compressing them into one agent-quality number.

A correct final answer does not prove the agent took the correct operational path.

The trajectory becomes part of the test evidence

The evaluators work from recorded trajectories in Strands Evals or OpenTelemetry traces in AgentCore Evaluations. AWS says skill invocations can be recognised from SKILL.md reads or native skill-loading tools across several agent frameworks. Per-skill results retain session, trace and span context, so a failed score can point back to the invocation that caused it.

This is more useful than reviewing output after the fact. A PTO agent can return a plausible response after checking balance while skipping rollover policy. The output may look acceptable; the trajectory shows the control failure.

Put skill regressions in the deployment pipeline

AWS recommends deterministic routing assertions for known requirements, judge-based selection and instruction-following evaluation, and calibration on each evaluation surface. It also supports on-demand, batch and continuously sampled production evaluation from traces.

The practitioner change is to treat a skill catalogue like executable policy. Add regression cases for critical routes, fail builds when mandatory skills do not load or follow their steps, and monitor production traces for catalogue drift after new skills are introduced. Keep end-to-end task quality as a separate measure. When routing and execution fail differently, they need different owners, fixes and release gates.

Written by the Devence Lab research team.