You cannot debug an agent from a latency graph

A conventional service tells you when it breaks. It returns a 500, latency spikes, the error rate crosses a threshold, someone gets paged. An entire generation of monitoring tooling is built on the assumption that failure is loud and that health is the absence of errors.

An agent fails silently and successfully. It returns 200. It responds in eight hundred milliseconds, comfortably inside its budget. Every span in the trace is green. It called three tools, all of which returned without error. And the answer it gave the customer was wrong, or ungrounded, or quietly outside policy.

There is no threshold on a latency graph that catches that. The system did exactly what it was asked and reported success, which is why so many teams discover agent failures through customer complaints rather than through the observability stack they are paying for.

What conventional monitoring is actually measuring

Application performance monitoring answers operational questions: did the request complete, how long did it take, which dependency was slow, what proportion of calls returned errors. Those questions are necessary and remain necessary. They are also, for an agent, close to the least interesting facts available.

The failure modes that matter in an agent are semantic. It retrieved the wrong document and reasoned faithfully over it. It called the correct tool with a subtly wrong parameter. It answered confidently from parametric memory when the retrieval returned nothing useful. It followed the user— and stopped following policy. Every one of those produces a clean trace.

Structure first: what the GenAI conventions get right

The industry has converged, sensibly, on OpenTelemetry as the substrate. The GenAI semantic conventions define how to represent a model call as a span, with attributes covering the model requested, input and output token counts, and the reason generation stopped. A trace links a user request to its retrieval steps, model calls, tool invocations and parsing, so a multi-step decision becomes a structure you can look at rather than a paragraph you have to imagine.

Adopt them, and adopt them early. The value is not that they answer the hard questions— they do not— but that they make the shape of a decision inspectable and portable across backends, and they do it in a schema that is standard rather than invented per team. Retrofitting instrumentation onto an agent that is already in production is considerably more painful than instrumenting it on the way in.

It is worth being precise about the boundary, though, because vendors are not always. Instrumentation records what happened. It does not judge whether what happened was correct, grounded, or safe. That judgement is a separate layer that scores the same spans against criteria you define, and no amount of tracing produces it for free. Teams that adopt the conventions and believe they have solved agent observability have solved the transport problem and left the evaluation problem untouched.

Three things worth recording that most teams do not

The first is the full decision input, versioned. Not the user’s prompt— everything that determined the output. The retrieved chunks and the index version they came from, the system prompt and its version, the tool schemas as they existed at that moment, the model identifier including any minor version the provider exposes. Six weeks later, when you try to understand an output, the retrieval corpus will have changed and the prompt will have been edited. If you did not capture what they were, you are guessing.

The second is the path not taken. Which tools were available and not called, which candidates retrieval returned and ranked below threshold, where the agent chose between branches. Failures are frequently about the option that was silently discarded, and a trace that records only the actions taken makes those failures invisible by construction. This is the difference between a log that tells you what the system did and one that tells you what it decided.

The third is termination context: why generation stopped, whether output was truncated, whether a tool timed out and the agent proceeded anyway. Truncation is a particularly nasty one, because a truncated response frequently still parses, still validates, and still looks like an answer— it is simply missing the qualification that came at the end.

The test is reconstructability, not visibility

The right standard to hold your telemetry to is not “can I see what happened” but “can I reconstruct why this specific output occurred, months later, for someone who does not trust me.” That is a much higher bar and it changes what you retain, for how long, and in what form.

It also changes retention economics, which is where this collides with reality. Full-fidelity agent telemetry is expensive— prompts and completions are large, and volume scales with usage. The instinct is to sample. Sampling is fine for latency percentiles and catastrophic for incident reconstruction, because the trace you sampled away is precisely the one you will be asked about. A workable compromise is to sample aggressively for aggregate metrics while retaining complete traces for anything that touched a consequential action, failed validation, or was escalated— and to decide which is which at instrumentation time rather than at query time.

Four questions your telemetry has to answer

A useful way to audit an instrumentation plan is to take a real output— ideally one that embarrassed you— and see whether the telemetry answers four questions without anyone consulting their memory.

What did the system decide, expressed as an action rather than a string? “Returned this text” is not a decision. “Approved a claim of this value under this policy” is. If your traces record generated tokens but not the consequential action they resulted in, you can debug the model and not the system.

On what basis? Every input that materially determined the output, at the version it held at that moment. If the answer involves re-querying a retrieval index that has since been reindexed, you cannot answer this question— you can only approximate it and hope the approximation holds.

What else was possible? The tools available and unused, the retrieved candidates ranked below the cut, the branch not followed. Without this you cannot distinguish a system that correctly rejected an option from one that never considered it, and those demand entirely different fixes.

Would it decide the same way now? This is the one almost nobody instruments, and it is the one that matters most for drift. It requires recording enough to re-execute the decision against the current stack and compare— and it is how you discover that a provider changed a model underneath you three weeks before your customers did.

Most teams can answer the first. Good teams answer the first two. The third and fourth are what separate telemetry that supports an incident review from telemetry that merely proves the system was running.

The part that becomes a data protection incident

One warning, because we have now seen this pattern more than once and it is expensive.

Prompts and completions are simultaneously the highest-value telemetry you can capture and the highest-risk. They contain whatever the user typed, which in a healthcare, financial or legal deployment means special-category personal data, account details and privileged material. Piping them wholesale into a third-party observability platform— one procured years ago for infrastructure metrics, under a data processing agreement written for latency numbers— is a disclosure nobody consciously approved.

Redact at the SDK, before the span leaves the process. Not at the backend, not in a scheduled cleanup job, not in the query layer. Once the data has crossed the boundary it has been disclosed, and deleting it afterwards changes your storage footprint rather than your legal position.

Where this ends up

Observability for agents is not a dashboard problem and it is not solved by buying a product. It is the practice of capturing, at the moment of a decision, enough context to defend that decision to someone hostile long after the system has changed underneath it.

Which is the same requirement as evidence. A team that instruments its agents this way discovers it has most of a decision log already— and a team that does not will find, during its first serious incident, that the traces it kept can prove the system was fast and cannot prove it was right.