AI Agent Observability: Follow the Cost of Real Work

Start with the business task, then trace its steps

A model dashboard can show activity while the business still has unresolved work. For a production agent, define a task as a specific outcome: a quotation saved, a service request assigned or a customer record updated. Give the task a durable identifier that remains available when processing continues in a queue or another service.

OpenTelemetry describes traces as collections of spans representing operations. Its tracing documentation also describes links between related spans, including asynchronous work. Use this technical structure to investigate a business task, while keeping the business identifier separately available for reconciliation. A trace is diagnostic evidence; the destination system remains the source of truth for whether the requested update exists.

AI agent observability tracing across a legacy boundary

Illustrative example: an agent prepares a quotation using an existing order system. The journey includes retrieving customer details, calling a model, checking inventory, saving a draft and returning its reference. Record where each step starts and ends, whether it failed, and how it connects to the original request.

The OpenTelemetry GenAI walkthrough demonstrates model and tool spans with model names and token counts. That is evidence of the documented example, not a promise that every agent framework exports the same fields. Inspect the instrumentation installed in your application and test the actual output.

If the legacy application cannot carry trace context, instrument the integration boundary and map its request or job reference to the workflow identifier. Label the unobserved interval explicitly. Do not attribute all of that elapsed time to model inference, and do not rewrite a critical system just to obtain a prettier trace.

AI agent cost management needs a complete ledger

Keep a cost ledger for every workflow attempt, including unsuccessful attempts. Record model usage, paid tool calls, attributable compute and the allocation method for shared infrastructure. Track operator intervention time separately; convert it into money only with an explicitly agreed rate. Show development and ongoing maintenance costs separately from variable runtime cost.

Avoid summing both a parent total and its underlying call records. Choose one authoritative level for each cost category, identify duplicate exports and retain the applicable pricing date. Where a provider offers aggregate billing but no per-request breakdown, mark the allocation as estimated and reconcile its total against the bill.

Use the same reporting window and task population for cost and outcomes. Keep unfinished tasks visible instead of treating them as successes or silently excluding their spend. When no valid task completed, report cost and zero completions rather than manufacturing a cost-per-success figure.

Collect enough evidence without copying every conversation

OpenTelemetry's sensitive-data guidance recommends minimizing collection and reviewing instrumentation output. Apply that principle to agent monitoring: prefer operational identifiers, durations, error classes and usage totals over full customer messages or tool payloads. Telemetry can also capture sensitive information by accident, so define access and retention for it.

For deeper debugging, use a deliberate, restricted capture process with sanitization. Keep a complete accounting record of usage and outcomes even if detailed diagnostic traces are sampled. A sampled trace set should not be treated as an exhaustive invoice or task register.

Turn the dashboard into a weekly engineering decision

Review where time and cost accumulate by workflow and result. If quotation drafts wait on inventory, reducing model tokens will not resolve that delay. If repeated missing-data requests drive spend, improve the intake form or validation. If people repeatedly repair the same output, investigate that pattern before expanding access.

Begin with one workflow, a responsible engineer and an agreed cost boundary. Add an investigation owner and an escalation route when a task exceeds its time or spend budget; preserve its unresolved state for follow-up. These are implementation requirements, not controls that appear automatically when tracing is enabled.

UVISION combines dedicated developers and managed projects with at least one full-time developer per project. For critical legacy systems, ongoing ownership means connecting AI capabilities to existing business rules, investigating production behavior and maintaining the integrations as the system evolves. The practical first deliverable is one workflow whose outcome, operating cost and unresolved work can be explained.