Evaluation, Testing & Red Teaming
Golden datasets, LLM judges, metrics that matter, testing strategy and red-teaming whole systems.
AI answers can differ each time, so test the qualities that matter, such as correct fields and allowed sources, instead of exact wording. Build a set of tests that includes tricky and hostile cases. Red teaming means trying to break your own system before others do.
- Golden dataset
- A set of trusted test examples.
- Red teaming
- Attacking your own system to find weaknesses.
- Calibration
- Checking that an AI judge agrees with human judges.
Traditional testing expects the same output every time. AI systems can produce several acceptable outputs. Their failures may depend on the wording, the context, the model version or the retrieved evidence. So evaluation becomes a product feature, not a final QA step. This lesson covers evaluation, LLM judges, metrics, testing strategy and red teaming.
Why AI evaluation is different
You need layers: fixed tests, search tests, model behavior tests, end-to-end task tests, safety tests and human review for selected cases. Build a golden dataset of typical inputs with the properties you expect. Do not test only the happy path. Include ambiguity, missing data, hostile instructions, broken documents, permission boundaries and long-context cases.
- Version datasets and evaluation criteria.
- Run evaluations before you change models, prompts, retrieval or tools.
- Treat evaluation as part of the product, owned by the team.
LLM as judge
An LLM can score outputs against a rubric. This is useful for qualities of meaning, such as relevance, completeness, style or being grounded. But judges are biased towards longer answers, confident language and their own model family. Use structured rubrics with criteria you can observe. Use pairwise comparisons, calibrated examples, fixed checks and regular human review. For factual tasks, require that the answer lines up with the evidence. Do not ask whether the answer sounds correct.
Score each criterion 0/1 with a one-sentence justification BEFORE the score. 1. GROUNDED every claim is supported by a cited source passage (quote the passage). 2. COMPLETE answers every part of the question. 3. POLICY does not promise anything outside policy P-12. 4. REFUSES declines and escalates when the sources do not contain the answer. Calibrate: have humans label 50 outputs; require >= 85% agreement with the judge before relying on it.
Metrics that connect to business results
- Search: recall@k, precision@k, MRR and nDCG.
- Classification: precision, recall, F1 and confusion matrices.
- Extraction: accuracy for each field.
- Generation: correctness for the task, being grounded, completeness and human preference.
- Operations: delay, token use, cost per successful task, tool error rate, retry rate, escalation rate and abandonment.
A senior AI engineer connects technical measures to business results. Answer quality improved 5 percent matters only if the improvement justifies the added cost, or improves an outcome that users care about. Track success for each completed task, not for each request. And analyze the trade-offs between cost and quality.
Unit, integration and end-to-end tests
Unit-test the fixed parts heavily: building prompts, schemas, authorization, tool wrappers, metadata filters and state changes. Integration tests check the boundaries between models and tools, and the search pipelines. End-to-end tests cover realistic tasks. Snapshot tests are useful, but they should not require exact prose. Assert properties instead. For example: it contains the required fields, it cites an allowed source, it chooses the expected tool, it refuses an unauthorized action, or it produces a valid workflow state. Keep a regression dataset for model behavior.
Red teaming agent systems
Red teaming tries to make the system break its intended behavior. It covers direct prompt injection, and indirect injection through documents and web pages. It covers tool abuse, data theft, privilege escalation, access across customers, endless loops, cost attacks and hostile tool responses. Automate the scenarios where you can. Each issue you find becomes a regression test and a design control. The architecture question is not can the model be tricked. Models can be influenced. The question is what a successful trick can make the system do. Limiting the impact is as important as improving the prompt.
- ✓Evaluation has layers: fixed tests, search, behavior, end-to-end, safety and human review. It needs a versioned golden dataset that includes hostile cases.
- ✓Check LLM judges against human labels, and combine them with fixed checks.
- ✓Measure cost per successful task, and tie measures to business results.
- ✓Assert properties, not exact prose. Red-team the whole system and keep the blast radius small.
Build a golden set and a red-team suite
- Write 40 cases for one feature: 20 typical, 8 edge cases, 6 sensitive to policy and 6 hostile. The hostile ones are an injection in a document, a request across customers, tool abuse and a cost attack.
- Write assertions about properties, not exact text. Run each case three times to measure consistency.
- Write a judge rubric with four criteria. Label 15 outputs by hand and work out how often you agree with the judge.
- Turn each failure into a permanent regression test and a named design control.
Q1Why assert properties and not exact output text?show answer
Several different outputs can be correct, so tests that need an exact match are brittle. Properties, such as valid fields, allowed citations, the expected tool and a correct refusal, test what really matters.
Q2Name two known biases of LLM judges.show answer
A liking for longer answers and confident language, and a preference for outputs from their own model family. Reduce them with rubrics, pairwise comparisons, calibration and human review.
Q3What is the key red-team question for an agent?show answer
Not can the model be tricked, but what can a successful trick make the system do. Then limit that impact with permissions, budgets and approvals.