20 min

Evaluating & Debugging Agents

Building an eval set, catching regressions, and debugging a broken agent loop.

Lesson 8 of 9

0 done

In plain English

Do not trust an agent because its answer sounds good. Test it with many examples, check the tools it used, and record every step so you can see what went wrong. Turn each real failure into a test so it cannot come back.

Evaluation (eval): A set of tests that show how well the AI does the job.
Trace: A step-by-step record of what the agent did.
Regression test: A test that stops an old bug from returning.

Picture it

Making an agent better

Test with many examples

Not one lucky answer

Record every step

The tools it used and why

Find the first wrong step

Walk backwards from the error

Turn the failure into a test

So it cannot come back

Then it goes back to step 1 and repeats

Using one AI to judge another is not enough by itself. A judge can be fooled the same way a person skimming an answer can. It decides the answer worked because it sounds believable, without checking whether the tool calls and the data really support it. So good testing has layers.

Why it looks right is not a test

An agent that acts on its own has tools, multi-step plans and handoffs. It can fail in ways you never see if you only check the final answer. Your testing has to grow with the agent's freedom. Correct tool use, the quality of the plan, each reasoning step, the handoffs between agents, safety, speed and cost each need their own checks.

A layered testing plan

Code checks on all traffic: is the tool call correct for its schema, is the JSON valid, are banned phrases absent, and do search-based answers stick to their sources? These are cheap and exact, and they run on everything.
An AI judge on a selection: flagged problems, sampled audits and deeper checks before launch. Not every request. It is expensive, and it is not perfect, as the warning below shows.
People review the highest-stakes sample. They are the last backstop for anything a judge model cannot be trusted to assess alone.

⚠A judge model can show hallucinated correctness. It rates an answer as a success because it reads well, without checking it against what the tools really did. If you want the judge to check the right thing, give it the trace: which tools were called and what they returned. Do not give it only the final answer.

Traces: what makes debugging possible later

Without a full trace, debugging a live failure means guessing from logs. A trace holds the inputs, the outputs, the prompt version, and every tool call and result, in order. Tools such as MLflow and LangWatch capture this for the whole workflow, and they can also test live traffic with the same judges and checks you use before launch.

minimal tracing — the shape

function trace(step: string, data: unknown) {
  spans.push({ step, data, ts: Date.now() });
  // in production: send to MLflow / LangWatch / your observability tool,
  // not just an in-memory array
}

trace("tool_call", { name: call.name, args: call.arguments });
trace("tool_result", { name: call.name, result });
trace("model_response", { content: res.content, usage: res.usage });

Build your test set from real failures

The most useful test set is not a list of cases you dreamed up. It is made of real failures you traced and turned into repeatable tests. Each time tracing catches a real bug, that trace becomes a permanent test. Then the same failure cannot quietly come back after your next change to a prompt or framework.

→Try it: add the minimal tracing above to the agent loop from the tool-calling lesson. Cause one of the four failures from that lesson on purpose. Check that the trace makes it clear what went wrong without running the agent again.

Going deeper: a working test harness

A test harness is small. It has a set of cases, a runner that runs the agent several times on each case, scorers and a report. Keep the scorers as plain code wherever you can. Use judge-model calls only where a rule cannot decide.

run_eval.py

import json, statistics as st

def run_eval(agent, cases_path, trials=3):
    report = []
    for case in map(json.loads, open(cases_path)):
        outcomes = []
        for _ in range(trials):
            trace = agent.run(case["input"], capture_trace=True)
            outcomes.append({
                "tool_ok":   check_tools(trace, case["expected_tools"], case.get("forbidden_tools", [])),
                "schema_ok": validate(trace.final, case["schema"]),
                "state_ok":  check_end_state(case["world_after"]),        # did the world change correctly?
                "steps":     len(trace.steps),
                "cost":      trace.cost_usd,
            })
        passed = [all(o[k] for k in ("tool_ok", "schema_ok", "state_ok")) for o in outcomes]
        report.append({
            "id": case["id"], "category": case["category"],
            "pass_rate": sum(passed) / trials,           # consistency, not a lucky pass
            "all_pass": all(passed),
            "median_steps": st.median(o["steps"] for o in outcomes),
            "cost": sum(o["cost"] for o in outcomes) / trials,
        })
    return report

# Summarize by category: overall all_pass rate, flaky cases (0 < pass_rate < 1), cost per success.

Task success: did the end result match the goal? Give this the most weight.
Tool accuracy: the right tool, valid inputs and no forbidden calls.
Efficiency: the steps and tokens compared with a good reference run. A sudden jump means loops.
Robustness: the pass rate over repeated tries, and under odd inputs such as typos, missing fields and hostile text.
Safety: rule breaks and resistance to injection should be zero-tolerance numbers.
Cost and speed for each successful task.

→Flaky cases, which sometimes pass and sometimes fail, are the most valuable ones. They show where the agent is fragile. Look into them before you add new cases. For the live side, such as release gates, canaries and sampling real traffic, continue to the Agent Ops path.

Key takeaways

✓Test tool use, planning, handoffs, safety, speed and cost, and not just the final text.
✓Use layers: code checks on everything, an AI judge on samples, and people on the highest stakes.
✓Give judges the trace. Otherwise they rate believable answers as correct.
✓Turn every real failure into a permanent test.

Hands-on exercise

From a trace to a test

Add the minimal tracing from this lesson to an agent: steps, tool calls, results and usage.
Cause one kind of failure on purpose and capture the trace.
Turn that trace into a test with code checks: the right tool was called, the schema is valid and an empty result is not accepted.
Fix the bug, run the test again, and keep it in a growing suite.

Check yourself

Q1What is hallucinated correctness when an AI is the judge?show answer

The judge rates an answer as a success because it reads well, without checking the evidence of what the tools really did.

Q2Which checks should run on all of the traffic?show answer

Cheap code checks: schema validity, correct tool calls, banned content and checks that answers stick to their sources.

Q3Why build tests from real failures?show answer

They match real user inputs and real bugs. They also stop the same failure from quietly returning after later changes.

→Next, continue to the Agent Ops & Governance path for the live versions of tracing and testing: trace formats, release gates, canaries and incident response.

← Multi-Step Planning, Memory & State Deploying to Production + Capstone →