Observability & Tracing

You cannot run what you cannot see. With ordinary software, a stack trace points at the bug. With an agent, the final answer is wrong and the cause is somewhere in a twenty-step run. Maybe a search returned the wrong document. Maybe a tool call had a slightly bad input. Maybe a hand-off dropped a limit. Observability for agents means recording the whole run, so that you can replay any run afterwards.

Traces, spans and the agent run

Borrow the model of distributed tracing. A trace is one run from start to finish, such as one user request. It holds spans. A span is one timed operation: a model call, a tool call, a search, a call to a sub-agent or a guardrail check. Spans sit inside each other. So you can see that the research span held five parallel worker spans, each with its own model and tool spans.

What to record on every span

• Identity: the trace ID, span ID, parent, session ID, user or customer ID (hashed or made anonymous where needed) and the agent name.
• Versions: the prompt version, the model name and exact snapshot, the tool schema version and your own release ID. It broke on Tuesday needs a version to point at.
• Inputs and outputs: the full prompt or messages sent and the reply received (after removing private data), plus tool names, inputs and results.
• Settings: temperature, maximum tokens, stop rules and any seeds.
• Numbers: input, output and cached tokens, delay, time to first token, retries and cost.
• Outcome: the status, the error type and message, the guardrail verdicts and the finish reason, such as stop, length or tool_calls.
• Feedback: thumbs up or down, edits, escalations and later review labels from people, linked back to the trace.

from opentelemetry import trace
tracer = trace.get_tracer("support-agent")

def traced_llm_call(messages, *, model, prompt_version, **params):
    with tracer.start_as_current_span("llm.call") as span:
        span.set_attribute("gen_ai.request.model", model)           # OTel GenAI semantic conventions
        span.set_attribute("app.prompt_version", prompt_version)    # your own attributes
        span.set_attribute("app.release", RELEASE_ID)
        try:
            resp = client.generate(model=model, messages=messages, **params)
            span.set_attribute("gen_ai.usage.input_tokens", resp.usage.input)
            span.set_attribute("gen_ai.usage.output_tokens", resp.usage.output)
            span.set_attribute("app.cost_usd", price(model, resp.usage))
            span.set_attribute("app.finish_reason", resp.finish_reason)
            return resp
        except Exception as e:
            span.record_exception(e)
            span.set_status(trace.Status(trace.StatusCode.ERROR))
            raise

def traced_tool_call(name, args):
    with tracer.start_as_current_span(f"tool.{name}") as span:
        span.set_attribute("tool.name", name)
        span.set_attribute("tool.args", redact(json.dumps(args))[:4000])
        result = TOOLS[name](**args)
        span.set_attribute("tool.result_preview", redact(str(result))[:2000])
        return result

Choosing tools

You can send OpenTelemetry data to any general tool. Or you can use a platform made for LLMs, which adds prompt viewers, links to testing and dataset management. Popular choices include Langfuse (open source, can be self-hosted), LangSmith, Arize Phoenix (open source), Braintrust, Helicone, Weights and Biases Weave, and the LLM parts of general monitoring vendors like Datadog. Some questions matter more than features. Can you host it yourself, or meet your data-location rules? Does it show long multi-agent traces in a readable way? Can you turn a bad live trace into a test case in one click? Can you export your data, so you are not locked in?

Privacy, removing data and keeping records

• Traces hold user prompts, which may contain personal data, secrets or confidential business content. Treat the trace store as sensitive as your database.
• Remove private data before you store it: emails, phone numbers, card numbers, API keys and tokens. Use pattern-based removal, plus a list of what is allowed to be logged in full.
• Set how long to keep data by purpose. Keep raw content for a short time, and totals for longer.
• Limit access by role, and log who viewed which traces.
• Honor deletion requests. You need to be able to find traces by user ID.

Dashboards and alerts that earn their place

• Key signals for each agent: number of requests, success rate, p50 and p95 delay, error rate and cost per run.
• Agent-specific signals: steps per run, tool-call error rate by tool, retry rate, how often the budget is hit, escalation rate and how often guardrails trigger.
• Alert on change, not only on thresholds: tokens per run jumping three times, a new top error type, or a sudden fall in tool success.
• Alert on how fast money is spent: dollars per hour against a ceiling, with an automatic circuit breaker for runaway loops.
• Keep alerts few and useful. Each one links to a runbook and a trace view that is already filtered.

Debugging a bad run: a routine you can repeat

• Start from the symptom. Which output or action was wrong?
• Walk the trace backwards to the first span where something went wrong: a fact, an input or a missing limit.
• Name the root cause: search, prompt, tool, model, hand-off or input.
• Reproduce it by replaying that span's exact input against the pinned model and prompt.
• Fix it. Then add the trace to your test set as a permanent regression case.