Site icon Decode the future

AI Agent Observability: Traces, Evals and Costs

ai agent observability og

Last updated: September 11, 2026. By Ignacy Kwiecien, founder and editor-in-chief, DecodeTheFuture.org.

Quick answer: AI agent observability connects traces, logs, metrics and evaluations so a team can explain an agent run, its decisions, latency, cost, failures, safety events and outcome quality.

Short answer: AI agent observability is the practice of recording enough structured evidence to explain what an agent did, why it chose each model or tool, how long the run took, what it cost, which failures occurred and whether the final result satisfied the task. A useful implementation combines traces, logs, metrics and evaluations. It does not mean storing every prompt forever or putting a larger dashboard in front of an unreliable system.

Metrics satellite: For concrete definitions of task success, latency, tool reliability, token usage and cost, see AI agent observability metrics.

Agents make ordinary application monitoring incomplete. A web request may call a model, ask a retrieval system for context, invoke several tools, retry a provider, hand work to another agent and still return HTTP 200. The response can look successful while the workflow used the wrong source, exceeded a cost budget or silently skipped a required step. Observability makes those hidden decisions visible enough to operate the system responsibly.

This guide focuses on production implementation. It explains the data model, the signals worth measuring, the relationship between tracing and evaluation, and a rollout that keeps sensitive payloads under control. For the wider system design, see the AI architecture for production guide. For benchmarking methodology, use the separate AI agent benchmarks overview; a benchmark score is not a substitute for observing your own workload.

What AI agent observability needs to answer

An observability system should answer five questions for a particular run: what task arrived, what path the agent took, what resources it consumed, what went wrong and whether the result was good enough. These questions must be answerable at different levels of detail. An operator needs a quick aggregate view, while an engineer investigating a bad answer needs to open one trace and follow its children.

  • Identity: Which user, workflow version, tenant, request and experiment produced the run?
  • Decision path: Which model, prompt version, retrieval query, tool and fallback was used at each step?
  • Performance: What were time-to-first-token, total latency, queue time, tool latency and retry delay?
  • Resource use: How many input and output tokens, GPU or provider units, tool calls and external requests were consumed?
  • Outcome: Did the workflow meet its contract, pass checks, require human review or fail in a way the user could see?

The word “agent” does not guarantee a particular architecture. A single model with tool calling, a graph workflow and a multi-agent coordinator can all be observed using the same principles. The important boundary is the run: one request or task should have a stable identity, and every meaningful operation beneath it should be related to that identity.

Traces, logs, metrics and evals are different signals

Observability becomes confusing when these terms are treated as interchangeable. They answer related questions but operate at different scales.

SignalBest questionTypical exampleMain limitation
TraceWhat happened during one run?Model call followed by retrieval, tool call and fallbackExpensive to retain at full detail
LogWhat event or error was emitted?Provider returned a rate-limit or validation errorRelationships can be lost without IDs
MetricHow is the system behaving over time?p95 latency, tool error rate or cost per successful taskAggregates hide individual causes
EvaluationWas the output or trajectory acceptable?Groundedness check, schema validation or human rubricA score depends on the dataset and evaluator

In practice, a metric should link back to representative traces, and a trace should point to the evaluation result that describes its outcome. Logs need the same correlation identifiers. This is the difference between a dashboard that reports “quality fell” and an operational system that can show which workflow version, provider, tool or document set caused the fall.

OpenTelemetry’s semantic-convention work is useful for choosing common names and meanings for telemetry fields. The OpenTelemetry semantic conventions documentation describes conventions for spans, metrics and attributes, while the GenAI attribute registry is a living reference for AI-related attributes. Treat those fields as a compatibility aid, not as permission to assume that every SDK exports every attribute or that every convention has the same stability level.

Build a trace tree for the agent run

The smallest useful trace is a tree rather than a flat list. Create a root span for the user-visible task and child spans for operations that have their own timing, errors or cost. A typical run may look like this:

  1. Agent run: the workflow name, version, tenant and task identifier.
  2. Planning step: the model and prompt version used to decide the next action.
  3. Retrieval step: query, index version, result count and retrieval latency, with sensitive text redacted.
  4. Tool call: tool name, validated argument shape, external request ID, result status and duration.
  5. Model generation: provider, model identifier, input/output token counts, finish reason and latency phases.
  6. Guardrail or evaluator: policy result, schema validation, groundedness check or human-review decision.

Do not create a span for every internal function. A span is valuable when it helps explain a user-visible decision, a reliability boundary or a measurable resource. Excessively fine-grained traces raise storage costs and make the trace harder to read.

Use stable identifiers at the boundaries. A browser request might have a request ID, the application a run ID, the distributed trace a trace ID and each provider call its own request ID. Store them as related fields. OpenAI documents request and error details in its API error guide; Anthropic documents typed errors and request identifiers in its API errors reference. The implementation should preserve those identifiers for support and debugging without exposing them to end users unnecessarily. If the workflow uses an SDK, the agent implementation guide is a useful adjacent reference for keeping execution boundaries explicit.

For asynchronous work, retain the parent-child relationship across the queue. An agent that starts a background retrieval or approval task must not appear as a collection of unrelated traces. Propagate a trace context or an equivalent run identifier through the message. If a workflow intentionally starts a new security or tenant boundary, record a link between the traces and the reason for the boundary.

A minimal event model that remains useful later

A practical event schema should be small enough for every service to emit consistently. Start with fields that support identity, timing, outcome and accounting. Payload capture is optional and should be governed separately.

Field groupExamplesWhy it matters
Identitytrace_id, run_id, parent_id, tenant_hash, workflow_versionConnects events and permits safe grouping
Operationspan_kind, model, provider, tool_name, route_reasonExplains what the agent attempted
Timingstarted_at, duration_ms, queue_ms, ttft_msSeparates model, tool and infrastructure delay
Usageinput_tokens, output_tokens, cached_tokens, estimated_costSupports budgets and unit economics
Outcomestatus, error_type, finish_reason, validation_resultDistinguishes success from an HTTP-level response
Governanceredaction_profile, retention_class, evaluator_versionMakes sensitive data and score changes auditable

Use enumerated values for important dimensions. A field such as status should not contain dozens of spelling variants. Keep the original provider error type in a separate field and map it to an internal class such as authentication, quota, timeout, validation, overload or tool failure. This permits cross-provider dashboards without throwing away provider-specific detail.

Record the reason for a route or fallback. “Used cheaper model” and “fell back after timeout” are different events, even if both end at the same model. The route decision may be based on task capability, latency budget, price, region, data policy or an experiment. A reason code helps explain why a cost or quality distribution changed after deployment.

Instrument the control flow, not only the model call

Many teams add an LLM span and call the system observable. That misses the parts that make agents difficult to operate: tool selection, loop termination, retrieval quality, handoffs and human approvals. Instrument the control flow around the model.

Tool calls

Record the tool name, version, argument validation result, start and end time, status, retry count and a bounded result summary. Never log credentials, authorization headers or unrestricted tool output by default. A failed tool should be distinguishable from a model that produced an invalid tool request.

Retries and fallbacks

A retry is not free success. Record the original failure, the backoff, the attempt number and whether the fallback changed the model or provider. Cap retries by a time and cost budget. If a fallback produces a lower-quality answer, the run outcome should preserve that fact rather than reporting only that the final HTTP request succeeded.

Loops and handoffs

Track loop depth, repeated tool arguments, agent handoffs and the reason an agent stopped. A loop-depth histogram often reveals a prompt or tool contract regression before users report an outage. Set a maximum depth and a maximum wall-clock budget. Treat hitting either limit as a visible outcome requiring review, not as an ordinary success.

Retrieval and structured output

For retrieval, record the collection or index version, query representation, number of candidates, filters and score distribution. For structured output, record schema version and validation result. Store the raw document or full response only when the data policy permits it. The observability record should make it possible to say “the response failed validation against schema v3” without retaining a customer’s entire payload.

Metrics that matter in production

Start with a small scorecard that joins reliability, quality, speed, cost and safety. A high-level success rate alone can hide a system that answers quickly but incorrectly.

  • Task success rate: the share of runs that satisfy an explicit task contract or pass a human-reviewed sample.
  • Step success rate: model, retrieval, tool and validation success measured separately.
  • Latency: p50, p95 and p99 total latency, plus time-to-first-token, queue time and tool latency.
  • Reliability: provider error rate, timeout rate, rate-limit rate, retry rate, fallback rate and incomplete-run rate.
  • Agent behaviour: average and p95 step count, loop depth, tool calls per task, handoffs and human escalation.
  • Economics: input/output tokens, cached tokens, cost per run and cost per successful task.
  • Safety and data quality: policy violations, redaction events, schema failures, unsupported claims and retrieval-without-citation events.

Use denominators that reflect user value. Cost per request can improve when the system fails early; cost per successful task reveals whether that is a genuine improvement. Likewise, average latency can look stable while p99 becomes unacceptable for a customer workflow. Segment dashboards by workflow version, model route, tenant class, geography and tool where the sample supports it.

Set alerts on changes in distributions, not only fixed thresholds. A new model may increase output length by 30%, raise p95 latency by 20% and preserve the error rate. That is still an operational change. Keep a release marker in the telemetry so an engineer can compare before and after without guessing when the deployment occurred. The related LLM inference cost comparison explains why token accounting and infrastructure assumptions must be kept separate.

Tracing shows the path; evaluations show whether it was good

Tracing can prove that a tool ran and a response returned. It cannot by itself prove that the answer was correct, useful or safe. Evaluations provide that second dimension. The strongest systems connect evaluation results to the trace that produced them.

Offline evaluation before release

Build a small, versioned dataset from representative tasks and known failure cases. Include easy cases, ambiguous requests, tool failures, long contexts, permission boundaries and cases that require the agent to refuse or ask a clarifying question. Store expected properties rather than pretending every answer has one exact string.

Use deterministic checks where possible: schema validity, required citations, allowed tool use, numerical tolerances and policy decisions. A model-based judge or rubric can help with open-ended quality, but its version, prompt, model and disagreement rate must be recorded. Never treat a judge score as ground truth without sampling human review.

Online monitoring after release

Sample traces for review based on risk and novelty. Sample more heavily after a model, prompt, retrieval index or tool version changes. Compare quality by route and workflow version, not only by the entire site. A stable aggregate score can hide a regression affecting one customer or one language.

Evaluation data should be connected back to the agent trace through an evaluation ID and evaluator version. That lets the team reproduce why a run was marked as a failure and re-score historical traces when the rubric is corrected. It also prevents a dashboard from mixing scores produced by incompatible evaluator versions.

Latency, fallback and cost observability

Agent cost and latency are properties of the whole trajectory. The simple model cost estimate is:

input tokens × input rate + output tokens × output rate + tool cost + infrastructure cost

The rates vary by provider, model, region and contract, so keep them in a dated pricing configuration rather than hard-coding them into article examples or dashboards. Cache reads, batch jobs, GPU time, vector search and external APIs can change the total even when token counts are identical. Report both estimated cost and the rate-card version used.

For latency, split the total into queue, time to first token, generation, tool, retrieval, serialization and client delivery. When the p95 rises, the trace should show which component moved. A fallback can reduce errors while increasing latency and cost; a faster model can reduce latency while lowering quality. The right unit is often “cost and latency per successful task,” segmented by the route chosen.

Use a budget policy for retries. For example, a workflow may allow one retry for a transient provider failure but no retry after a schema error unless the prompt or arguments change. The rule belongs in the trace as a decision, so later analysis can distinguish a designed fallback from uncontrolled repetition.

Privacy, security and retention

Agent telemetry can be more sensitive than ordinary application logs because it may contain user requests, retrieved documents, credentials embedded in tool output or proprietary reasoning context. The safest default is to record metadata and bounded summaries, not full payloads.

  • Redact secrets and personal data before export, with tests for common credential patterns.
  • Separate identifiers from payloads and hash or tokenize tenant/user identifiers where exact identity is not needed.
  • Apply retention classes: short retention for raw payloads, longer retention for aggregate metrics and redacted failure signatures.
  • Restrict trace access by tenant and role; a support engineer should not automatically see another customer’s prompts.
  • Log redaction, sampling and access decisions so missing data is explainable.
  • Document where telemetry is stored and which providers receive it.

OWASP’s Top 10 for LLM Applications and NIST’s AI Risk Management Framework are useful risk references, but neither removes the need for a workload-specific data policy. If a team cannot explain who can inspect a trace, it is not ready to capture unrestricted prompts in production.

A practical production architecture

A robust design separates application instrumentation from storage and analysis. The application emits spans and structured events through a collector or telemetry gateway. The gateway applies sampling, redaction and routing rules. A trace store supports investigation, a metrics store supports alerting, and an evaluation store keeps datasets, scores and reviewer decisions. A cost service joins usage with dated rates.

The control plane should also contain release markers, workflow versions, prompt versions, model aliases, tool schemas and evaluator versions. Without those dimensions, teams are forced to compare “before and after” using wall-clock dates and memory. The data plane should remain resilient: if telemetry storage is briefly unavailable, the agent should follow a defined fail-open or fail-closed policy rather than blocking unpredictably.

A useful dashboard has three views:

  1. Executive: successful tasks, cost per success, quality sample, p95 latency and material safety events.
  2. Operator: error classes, provider health, fallback rate, queues, tool failures and current releases.
  3. Engineer: trace search, span waterfall, request IDs, payload-safe inputs/outputs and linked evaluations.

Do not build the dashboard first. First instrument one high-value workflow end to end, verify that an engineer can reconstruct a failure, and only then standardize the event contract for other workflows. This makes missing fields visible before the system has produced millions of traces. When the workflow is ready to compare providers, the inference API guide provides a separate decision context; it should not be confused with observability itself.

Rollout plan and common mistakes

A controlled rollout can be completed in stages:

  1. Define the task contract: write down what counts as success, refusal, escalation and incomplete execution.
  2. Instrument the root trace: add run identity, workflow version, outcome and total latency.
  3. Add child spans: model, retrieval, tool, validation, retry and handoff operations.
  4. Add accounting: tokens, rate-card version, tool cost and cost per successful task.
  5. Connect evaluations: deterministic checks first, sampled human review and versioned judges second.
  6. Set privacy controls: redaction, retention, access, sampling and incident procedures.
  7. Alert and learn: monitor distributions, investigate traces, update the dataset and feed recurring failures back into tests.

Common mistakes are predictable. Recording only the final answer loses the decision path. Recording only raw payloads creates a privacy and storage problem. Treating model-provider success as task success hides tool and quality failures. Using one average latency number hides tail behaviour. Changing prompts, models and evaluators in one release makes attribution difficult. Finally, an evaluation that never samples real production traces will drift away from actual user behaviour.

Observability also needs an ownership model. Someone must own the event schema, someone must review alerts, and someone must decide when a quality regression blocks release. A dashboard without an operational decision attached is documentation, not control.

FAQ

What is the difference between AI agent monitoring and observability?

Monitoring usually means watching known indicators and alerting when they cross a threshold. Observability is broader: it provides enough correlated evidence to investigate an unfamiliar failure. Monitoring is part of observability, but a task-success metric without traces, IDs and outcome context is not sufficient for agent operations.

Do I need to store complete prompts and responses?

No. Start with metadata, redacted excerpts, hashes, schemas and failure summaries. Full payload retention should be an explicit decision based on data sensitivity, access controls, retention limits and debugging value. Many incidents can be diagnosed from the route, tool arguments’ shape, error type, timing and evaluator result.

Which metric should I implement first?

Implement a stable task identity and outcome first, then total latency, error class, token usage and trace links. A metric such as cost per successful task becomes meaningful only after success is defined. Add step-level and quality metrics once the root trace is reliable.

Is OpenTelemetry mandatory?

No. An equivalent internal schema can work. OpenTelemetry is attractive when a team wants portable trace concepts and integration with existing telemetry infrastructure. The important requirement is consistent correlation and documented field semantics. Do not adopt a library while leaving provider request IDs, tool spans or redaction policy undefined.

How do I observe an agent that uses several model providers?

Use one internal run and route vocabulary, then store provider, model, attempt number and route reason on each model span. Normalize broad error classes while preserving the original provider error. This makes fallback rate, latency and cost comparable without pretending that providers have identical semantics.

Can an evaluation score be used as a quality KPI?

Yes, if the dataset, rubric, evaluator version and sampling method are explicit. Treat the score as one measurement with uncertainty, not as an absolute truth. Pair it with deterministic checks, production outcomes and human review of a risk-based sample.

What to remember

AI agent observability is a connected evidence system: one run identity, a trace tree, structured errors, measurable resource use and an outcome that says whether the task succeeded. Start with the workflow that matters most, keep payload capture conservative, and link aggregate alerts back to representative traces and versioned evaluations. That foundation supports reliable routing, cost control and safer improvements without turning every request into an unreviewable data dump.

Exit mobile version