18 min

Evaluating RAG Systems & Agents

Score each step of a retrieval pipeline and each part of an agent run, and read traces.

Lesson 6 of 10

0 done

In plain English

Systems with search or agents have several steps, and any step can fail. Score each step, then the whole. For search, check context recall and groundedness. For agents, check task success, tool choice, efficiency and safety, read the traces, and run each case several times.

Groundedness: Whether the answer's claims are supported by the sources.
Trace: The full record of an agent run, step by step.
Task success: Whether the agent reached the goal.

Picture it

Score each step of a retrieval system

Right documents found?

Select passages

Enough and relevant?

Write the answer

Supported by the passages?

Answer quality

Correct, complete, clear?

Refusal

Says unknown when it should?

Single-answer systems are the easy case. Most real products today use retrieval, tools or agents. They have several steps, and a failure in any step can ruin the result. This lesson shows how to evaluate each part of a retrieval system and an agent, so you can tell where a failure started.

Evaluate the parts, then the whole

Think of a question-answering system with retrieval as a pipeline. A question comes in. The system searches. It picks passages. The model writes an answer. Each step can fail. If you only grade the final answer, you will not know where to fix things.

rag-eval-map.md

STEP                 QUESTION TO ASK                            HOW TO CHECK
Understand question   Did it grasp what was asked?               Review a sample; rewrite checks
Search                Did the right documents come back?          Recall at k, hit rate on must-find docs
Select passages       Are the passages relevant and enough?       Judge: does the context contain the answer?
Write the answer      Is it faithful to the passages?             Judge: every claim supported (groundedness)
Answer quality        Does it answer the question, clearly?       Rubric: correct, complete, clear
Refusal               Does it say "I don't know" when it should?  Negative cases in the test set

The key RAG measures

Context recall: does the retrieved material contain the facts needed?
Context relevance: how much of the retrieved material is useful and how much is noise?
Groundedness or faithfulness: are the claims in the answer supported by the retrieved material?
Answer relevance: does the answer address the question?
Citation accuracy: do the cited sources really say what the answer claims?

A system can find the right document and still give a wrong answer. It can also give a right answer from memory, without the document, which will fail when the facts change. Separate scores show which is happening.

Evaluating agents

An agent chooses steps in a loop. It may call tools, read results and decide what to do next. You must check the result and the path.

Task success: did it reach the goal? Judge the final state, such as the ticket created or the right data returned.
Tool choice: did it pick the right tool at the right time?
Tool arguments: were the inputs correct and valid?
Efficiency: how many steps, how many tokens, how much time? Did it loop or repeat?
Safety: did it stay inside its limits and ask for approval when needed?
Recovery: when a tool failed, did it handle it sensibly?

Look at the trace

A trace is the full record of an agent run: each model call, tool call, input and output. Reading traces is the heart of agent evaluation. Check a sample of successes as well as failures. A run can reach the right answer by luck or by a risky path.

Make agent tests repeatable

Use a test environment with fake or copied data. Never test against real customer accounts.
Set up the starting state the same way each time.
Mock slow or costly tools where you can, and record real tool responses for replay.
Run each case several times, because agents vary. Report how often it succeeds, not just one run.
Put hard limits on steps and cost during tests.

⚠A single successful run proves little. Agents are not consistent, so measure success over several runs of the same case.

→Start by labeling the first wrong step in each failed trace. The pattern of first wrong steps tells you what to fix.

Key takeaways

✓Evaluate each part of a pipeline, then the whole. Otherwise you will not know where a failure began.
✓For retrieval, track context recall and relevance, groundedness, answer relevance and citation accuracy.
✓For agents, check task success, tool choice and arguments, efficiency, safety and recovery, and read the traces.
✓Make agent tests repeatable with safe test data, and measure success over several runs.

Hands-on exercise

Trace a failure to its step

Choose a retrieval-based assistant, or build a tiny one over five documents.
Write ten questions with the document that should be used. Include two where the answer is not in the documents.
For each question, mark: right document found, answer grounded, answer correct, refusal correct.
Count failures by step. Write which step to fix first and why.

Check yourself

Q1Why score search and answer separately?show answer

A failure can start in search or in writing. Separate scores show which step to fix.

Q2What does groundedness measure?show answer

Whether every claim in the answer is supported by the retrieved material, instead of being invented or recalled from memory.

Q3Why run each agent test case several times?show answer

Agents vary from run to run. Success over several runs is more honest than one lucky or unlucky result.

← Model Judges & Human Review Error Analysis & Debugging Quality →