Site icon Decode the future

AI Agent Observability Metrics: What to Measure

Last updated: September 11, 2026. By Ignacy Kwiecien, founder and editor-in-chief, DecodeTheFuture.org.

Short answer: production AI agent observability should measure the whole run, not only the model call. Start with task outcome, end-to-end latency, model and tool-call failures, token usage, estimated cost, retries, handoffs and a link to the trace. Add quality and safety signals with explicit sampling and versioning. The useful question is not “how many requests succeeded?” but “which work was completed, at what cost, with what latency and with what evidence?”

AI agents make familiar application metrics harder to interpret. A web request that returns HTTP 200 may still produce the wrong answer, call the wrong tool, repeat an expensive action or hand the task to a human after several hidden retries. Conversely, a run that contains a provider error may still be successful if a documented fallback completed the task safely. That is why a dashboard containing only uptime, average latency and token counts is not an observability system.

This article focuses on the narrower question behind the wider AI agent observability guide: which measurements should an engineering team define, how should they be calculated, and where should each signal live? The recommendations are vendor-neutral. OpenTelemetry is a useful interoperability layer, but the design principles still apply if your application uses another telemetry stack.

What makes an AI agent metric useful?

Direct answer: an agent metric is useful when it maps to a decision, has a stable denominator, can be connected to a representative run, and is segmented by the dimensions that change the result. A number without an outcome definition, time window or trace link may look precise while hiding the real failure mode.

Before collecting dozens of signals, define the unit being measured. A run can mean one user task, one workflow execution or one autonomous attempt, but the definition must be stable. A single run may contain several model requests, tool calls, retries and handoffs. If a team counts model requests as user tasks, its success rate and cost per task will be wrong as soon as the agent becomes more capable or more complex.

Keep aggregate metrics separate from detail: bounded labels such as model, route and error class belong in metrics; prompts, IDs and payloads belong in governed traces or logs. This follows the OpenTelemetry observability primer.

The core metric model: outcome, work and evidence

Direct answer: the minimum production model has three layers: outcome metrics describe whether the user task was completed; work metrics describe what the agent did; evidence links the result to traces, logs and evaluations. All three are needed because an outcome alone is not diagnostic and telemetry alone does not prove usefulness.

LayerExamplesQuestion it answersTypical home
Outcometask success, abandonment, human handoff, accepted answerDid the requested work finish acceptably?Metrics plus outcome event
Workmodel calls, tool calls, retries, tokens, queue timeWhat resources and steps did the run consume?Metrics and spans
Reliabilitytimeouts, rate limits, invalid arguments, provider errorsWhere did execution become unreliable?Metrics, events and error spans
Qualitydeterministic checks, evaluation score, user correctionWas the result correct and useful?Evaluation store linked to run
Safetypolicy event, prompt-injection detection, redaction outcomeWas the result produced within the operating boundary?Restricted events and review queues

These layers should share a correlation ID. The ID does not need to expose the user or prompt; it only needs to let an authorized engineer move from an aggregate point to the relevant trace. If the agent crosses services, propagate context across the boundary rather than creating a new unrelated ID for every model request. OpenTelemetry documents context propagation as the mechanism that connects work across process and service boundaries.

1. Task success and outcome quality

Direct answer: task success rate should be the percentage of eligible runs that meet the declared success criteria, not the percentage that returned a response. Define the criteria per workflow, record partial success separately, and retain the reason for failure so the number can drive an engineering action.

The simplest formula is successful eligible runs / eligible runs. The denominator must exclude only documented cases, such as a user cancelling before the agent begins. It should not silently exclude provider failures, tool timeouts or evaluator disagreements. If a workflow has multiple outcomes, use a bounded vocabulary such as success, partial, failed, abandoned and human_handoff. A single binary percentage can hide a rise in partial completions that users experience as failure.

Outcome quality needs a second measurement path. For a data-extraction agent, deterministic validation can check required fields, data types and reconciliation totals. For a research agent, citation presence and source-domain rules can be automated, while factual correctness may require a labelled sample. For a support workflow, a user correction, reopen event or human override can provide an online signal. None of these should be presented as a universal truth; each measures a defined proxy.

2. Latency: measure the path, not only the average

Direct answer: measure end-to-end latency together with queue time, model time, tool time, retry time and handoff time. Use p50 for the typical case and p95 or p99 for user-facing tail behavior. Always name the population, time window and completion state attached to the percentile.

End-to-end latency starts when the application accepts the task and ends when it returns a result or records a handoff. It may include authentication, retrieval, queueing, several model calls, tool execution, validation and rendering. Model latency is only one component. An agent can have a fast model and a slow experience because the tool waits on a remote database or because the route performs three sequential calls.

Streaming systems need at least two additional measurements: time to first token and time to final usable answer. A fast first token can improve perceived responsiveness while the agent continues tool calls in the background. If the user cannot act until the final answer is complete, final latency remains the operational metric. Measure both only when the product actually exposes streaming; do not use time to first token as a substitute for task completion.

MeasurementUseful breakdownCommon mistake
End-to-end durationworkflow, version, outcome, regionreporting only the mean
Model durationprovider, model, attempt, input/output modeincluding queue or tool time in the model span
Tool durationtool name, operation class, success/failureusing raw tool arguments as a metric label
Queue durationqueue, priority, load bandblaming the model for capacity wait
Tail latencyp95/p99, eligible completed runsmixing abandoned and completed runs without explanation

3. Tool-call reliability and agent behavior

Direct answer: tool-call observability should measure whether the agent selected the right tool, produced valid arguments, received a useful result and completed the call within its budget. Count attempts and logical calls separately, because retries can make a failing tool look busy rather than unreliable.

Useful measures include tool selection rate, argument-validation failure rate, authorization failure rate, timeout rate, application error rate, retry rate and successful result rate. A tool may return HTTP 200 while producing an empty or stale result, so the application should record a semantic validation outcome when one exists. A database lookup may be technically successful but unusable because the expected record was missing; that belongs in a bounded result classification.

Use one logical call ID across retries. The first attempt, second attempt and fallback should remain visible as separate spans under the same run. Otherwise a retry policy can inflate apparent tool volume, hide the original error and make cost analysis inaccurate. For side-effecting tools, retry only with an idempotency strategy or a domain-specific confirmation. Observability is not permission to repeat payments, bookings or destructive operations.

4. Tokens, cost and budget adherence

Direct answer: track input and output tokens per logical run and per model attempt, then calculate estimated cost from a versioned price table. Include retries, cached tokens where relevant, tool-related model calls and fallback routes. A token count is a usage measure; it is not by itself a bill and should not be presented as an exact invoice.

At minimum, record input tokens, output tokens, total tokens, model identifier, provider, request type and an estimate version. If a provider exposes cached-input or batch pricing, retain those categories separately. If the provider does not return usage, label the estimate and record how it was derived. Never silently fill a missing usage field with zero. Missing data should appear as unknown, because zero would make a route look cheaper than it is.

Useful business-facing measures include cost per successful task, cost per 1,000 eligible runs, cost by workflow, cost of failed runs and cost of retries. The denominator matters. Cost per model call can reward a route that requires many calls. Cost per successful task exposes whether a cheap model produces extra validation work or human handoffs. A team comparing providers should also inspect the LLM inference cost model rather than copying a headline token price into a forecast.

5. Errors, retries, fallbacks and provider health

Direct answer: classify failures by the layer that owns the next action: client input, authentication, rate limit, provider availability, timeout, tool contract, policy decision or application defect. Keep the original error and the final outcome. Retries and fallbacks should be visible as decisions, not disappear inside a generic “request failed” count.

One useful taxonomy starts with transport and infrastructure errors, then adds provider response errors, tool errors and semantic failures. A rate limit can call for backoff or routing; an invalid tool argument can call for schema correction; a policy refusal may be the correct outcome and should not be counted as an outage. The taxonomy must remain small enough to operate. A hundred labels that no engineer uses is worse than ten stable classes with trace evidence.

Report retry rate as runs with at least one retry / eligible runs and attempt count as a separate histogram. If the system retries every attempt, count the final failure and the retry cause. An increase in retries may precede an availability incident, but it can also come from a deliberately more conservative policy. Add route reason, retry budget remaining and fallback reason to the trace so the team can distinguish policy behavior from provider instability.

Provider health should be segmented by region, model, endpoint and error class only when those dimensions are reliable. Do not build a routing decision from a short-lived spike in one dashboard. Use a rolling window, minimum sample count and a recovery rule. When the fallback changes model capabilities, context limits, privacy terms or output format, record that fact and test the downstream contract.

6. Quality and evaluation metrics belong beside telemetry

Direct answer: evaluate the agent’s output with a combination of deterministic checks, curated test sets, sampled human review and model-based graders where appropriate. Store the evaluator version, rubric, dataset and sample policy with the score. An evaluation score is evidence about a defined task, not a universal quality percentage.

Use deterministic checks whenever possible. A structured-output validator, permission check, citation requirement or arithmetic reconciliation can be easier to interpret than a free-form judge. Model-based evaluation can add coverage for writing quality or complex answers, but store the judge model, prompt and rubric. For important decisions, sample human review and measure inter-reviewer disagreement. A falling judge score with stable human outcomes may indicate evaluator drift rather than product regression.

Quality should be sliced by route and fallback. If the primary provider fails and the backup produces a valid but less complete answer, the system needs a measured trade-off rather than an assumption that any successful HTTP response is equivalent. This makes evaluation a partner to agent workflow design, not a separate research exercise.

7. Safety, privacy and data handling metrics

Direct answer: safety observability should measure policy events, blocked actions, prompt-injection signals, sensitive-data handling and human escalation without collecting more content than the team is authorized to retain. The safest dashboard is not the one with the most prompt text; it is the one that supports an investigation with bounded, governed evidence.

Useful aggregate signals include policy intervention rate, sensitive-data redaction rate, tool calls blocked by authorization, prompt-injection detections, safety-review queue size and time to review. Define whether the metric counts attempts or unique runs. A single run with five blocked attempts should be one affected run for user-impact reporting but five events for control-plane capacity planning.

Payload capture needs its own metadata: capture mode, redaction status, retention class and access policy. OpenTelemetry’s current GenAI registry includes useful names for operations, tools and token usage, but it also warns that input and output messages may contain sensitive information. Align the implementation with the current GenAI semantic-convention registry, pin the version and review changes before adopting new attributes.

Never put raw prompts, responses, API keys or personal data into a high-cardinality metric label. Prefer a redacted summary, content hash, policy category or reference to a controlled record. If an incident requires payload inspection, access should be explicit and auditable. Privacy controls are part of observability quality because an instrumented system that cannot legally or safely be operated is not a successful production system.

8. Implementing a trace and metric schema with OpenTelemetry

Direct answer: model one user task as a parent trace and represent orchestration, model calls, tool calls, retrieval and validation as meaningful child spans. Add bounded attributes for workflow, provider, model, operation, attempt and outcome. Use metrics for aggregation and traces for the causal path; do not emit a span for every trivial function.

A framework-neutral trace might look like this:

{
  "run_id": "internal-opaque-id",
  "workflow": "invoice-review",
  "agent_version": "2026-09-11.3",
  "outcome": "partial",
  "spans": [
    {"kind": "agent", "duration_ms": 8420, "status": "ok"},
    {"kind": "model", "provider": "provider-a", "model": "model-x", "input_tokens": 1850, "output_tokens": 420},
    {"kind": "tool", "name": "invoice_lookup", "attempt": 1, "status": "timeout"},
    {"kind": "model", "provider": "provider-b", "route_reason": "fallback_timeout", "status": "ok"}
  ]
}

This is illustrative data, not a prescription for a provider SDK. The key is the relationship: one run, a trace tree, a visible failed attempt and a visible fallback. OpenTelemetry’s trace API describes spans as named units of work with timing, context, attributes, events and status. In practice, keep the attribute vocabulary documented and avoid adding arbitrary fields from every framework.

Semantic conventions help different services use common names, but they do not remove the need for local decisions. Pin the OpenTelemetry package and semantic-convention version, record the exporter configuration, and test whether the chosen backend preserves links and exemplars. If a convention is marked experimental or moved, treat upgrades as schema changes. The application’s stable internal run and outcome fields should not disappear just because a vendor attribute changes.

9. Dashboard design: a small set of views with a clear action

Direct answer: create separate views for user outcomes, execution health, cost, quality and safety, then connect each aggregate chart to example traces. One giant dashboard encourages teams to read correlated symptoms as independent failures. A smaller set of decision-oriented views makes an incident easier to triage.

Outcome view

Show success, partial completion, abandonment and human handoff over time. Break down by workflow version, input class and route when sample sizes support it. Add the denominator and sample traces so the owner can tell whether a release improved the task, not merely reduced errors.

Execution view

Show end-to-end p50/p95/p99, model and tool duration, queue time, step count, retry rate and timeout rate. Put provider and region filters next to the charts. Include a trace waterfall for a slow run and a normal run. This makes sequential tool calls, fan-out patterns and fallback chains visible without forcing an engineer to reconstruct them from logs.

Economics and quality view

Show cost per successful task, token distribution, budget exceedance, evaluation score and deterministic-check failures. Keep price-table version and evaluator version visible. A model route with low cost but weak quality should not appear as a winner just because the dashboard combines incompatible numbers into one score.

10. A practical rollout sequence

Direct answer: instrument one important workflow end to end before expanding to every agent. First establish run identity and outcome, then add timing and failure spans, then tokens and cost, and finally quality and safety sampling. This order produces useful evidence early while keeping the schema reviewable.

  1. Define the run. Write the start, end, eligible population and bounded outcome vocabulary. Agree on what counts as success and partial completion.
  2. Connect the trace. Propagate one correlation context through the orchestrator, model gateway, retrieval service and tools. Verify that an engineer can find the whole path.
  3. Measure the expensive and slow paths. Add end-to-end duration, queue time, model spans, tool spans, retry count, timeout class and token usage.
  4. Protect the data boundary. Redact payloads, remove secrets, cap attribute sizes and document retention. Test that raw content cannot become a metric label.
  5. Validate with examples. Compare a normal successful run, a tool timeout, a provider rate limit, a fallback and a user handoff. If the trace cannot explain them, the schema is not ready.
  6. Add evaluation. Link deterministic checks and sampled quality review to the same run ID. Version the rubric and report missing evaluator data explicitly.
  7. Turn evidence into action. Add only alerts whose owner, threshold, window and response are documented. Review the metric catalogue after the first incident.

This is also the point where teams should compare instrumentation cost against value. High-volume traces can be sampled, while errors, fallbacks and budget violations may deserve a higher retention rate. Sampling rules must preserve enough information to estimate rates and must not selectively hide a particular provider or user segment.

Common measurement mistakes

Direct answer: most observability failures come from ambiguous denominators, missing run-level outcomes, unbounded labels, hidden retries and treating quality as a transport property. Fix the model before adding another dashboard panel.

  • Counting HTTP 200 as success. A response can be syntactically valid and operationally useless. Emit an application-level outcome.
  • Using average latency. Average values hide slow tails and do not describe streaming or handoffs. Use distributions and clear populations.
  • Mixing calls and runs. Model requests, logical tool calls and user tasks have different denominators. Keep them separate.
  • Hiding retries. A successful fallback may be good user experience but still important infrastructure evidence. Record both.
  • Assuming token usage equals cost. Prices, cached tokens, contract terms and missing usage fields change the estimate.
  • Publishing a single quality score. Keep deterministic checks, sampled review, evaluator version and uncertainty visible.
  • Capturing everything forever. Full payload retention increases privacy, security and cost risk. Start with the minimum evidence needed.
  • Using high-cardinality metric labels. Put IDs and detailed content in governed traces or logs, not aggregate time series.

The aim is not perfect telemetry. It is a system in which a team can explain a material change in outcomes, cost or latency and decide what to do next. The practical LLM observability tools comparison belongs to a separate page.

FAQ

What is the most important AI agent observability metric?

Start with task outcome rate linked to a trace. It tells you whether the workflow delivered the intended result and gives engineers a path to inspect latency, tools, model calls and errors. The exact success rule must be defined for the workflow.

Which latency metrics should an AI agent track?

Track end-to-end duration, p50, p95 and p99, then decompose the path into queue, model, tool, retry and handoff time. For streaming products, add time to first token and time to final usable answer.

How do you measure AI agent cost?

Record input and output tokens per model attempt, provider and model, then calculate an estimate from a versioned price table. Report cost per successful task and include retries and fallback calls. Missing usage is unknown, not zero.

Are AI agent observability metrics the same as LLM metrics?

They overlap but are not identical. LLM metrics describe model calls, while agent metrics also need workflow outcomes, tool-call behavior, step count, handoffs, retries and the final task result.

Is OpenTelemetry required for AI agent observability?

No. OpenTelemetry provides portable concepts and conventions for traces, metrics and attributes, but another documented telemetry schema can work. The non-negotiable parts are stable correlation, useful outcomes, bounded data and traceable evidence.

Should prompts and responses be stored in traces?

Not automatically. They may contain personal, confidential or secret data. Start with metadata, redacted bounded summaries, hashes and failure signatures. Full payload capture should be an explicit, access-controlled and retention-limited decision.

What to remember

AI agent observability becomes actionable when the run has an explicit outcome and every important step can be explained. Measure success, decompose latency, separate calls from retries, calculate cost from visible usage, and version quality and safety evidence.

A first release needs a run ID, outcome, trace, latency distribution, tool reliability, token usage, error class and cost estimate. Later, evaluations and safety measures can grow around decisions. This page strengthens the production observability framework instead of duplicating it.

Sources

Research and links checked 2026-09-11. Provider implementations, prices and semantic conventions can change; verify the version used by your stack before implementation.

Exit mobile version