Become an AI evaluation engineer

Honest note: The ten steps follow the AI Evaluation Engineer course. Extra reading from other courses is listed at the end. It does not count towards the certificate.

1. 1

   Step 1 of 10

   Understand the role and the loop

   Explain why AI needs its own testing, and what an evaluation engineer does.

   Lessons0/1 done

   • ○What an AI Evaluation Engineer DoesCourse: AI Evaluation Engineer

   Before you move on: Read 50 real AI outputs, mark them, and group the bad ones into types.

2. 2

   Step 2 of 10

   Define what good means

   Turn a vague wish into a rubric that two reviewers can apply the same way.

   Lessons0/1 done

   • ○Defining Quality & Writing RubricsCourse: AI Evaluation Engineer

   Before you move on: Write a rubric with three dimensions and two must-never-happen rules, and test it with a friend.

3. 3

   Step 3 of 10

   Build a test set

   Collect real, varied cases with right answers, and keep a hidden set.

   Lessons0/1 done

   • ○Building Test Sets & Golden DataCourse: AI Evaluation Engineer

   Tool: BraintrustTool: LangSmith

   Before you move on: Build 30 cases across five groups, split into development and hidden sets.

4. 4

   Step 4 of 10

   Add automatic checks

   Use code to check shape, matches and search quality.

   Lessons0/1 done

   • ○Automatic Checks & MetricsCourse: AI Evaluation Engineer

   Before you move on: Write five code checks and work out precision and recall for a small example.

5. 5

   Step 5 of 10

   Use judges and human review

   Score open qualities, and check every judge against people.

   Lessons0/1 done

   • ○Model Judges & Human ReviewCourse: AI Evaluation Engineer

   Before you move on: Compare a model judge with your own scores on 20 outputs and fix one pattern.

6. 6

   Step 6 of 10

   Test retrieval systems and agents

   Score each step of a pipeline and read agent traces.

   Lessons0/1 done

   • ○Evaluating RAG Systems & AgentsCourse: AI Evaluation Engineer

   Tool: LangfuseTool: Arize Phoenix

   Before you move on: For ten questions, mark which step failed first and count failures by step.

7. 7

   Step 7 of 10

   Study the failures

   Find root causes and fix one thing at a time.

   Lessons0/1 done

   • ○Error Analysis & Debugging QualityCourse: AI Evaluation Engineer

   Before you move on: Run an error analysis on 30 outputs and write one targeted fix with a before and after score.

8. 8

   Step 8 of 10

   Gate every release

   Stop changes that make quality worse, and roll out safely.

   Lessons0/1 done

   • ○Regression Tests & Release GatesCourse: AI Evaluation Engineer

   Before you move on: Write a release gate with five rules and a rollout plan with a rollback step.

9. 9

   Step 9 of 10

   Watch it in production

   Sample live traffic, catch drift and feed failures back into the test set.

   Lessons0/1 done

   • ○Production Monitoring & Feedback LoopsCourse: AI Evaluation Engineer

   Calculator: Agent run cost calculatorTool: HeliconeTool: OpenTelemetry

   Before you move on: Write a monitoring plan with two alerts and a weekly ten-conversation review.

10. 10

    Step 10 of 10

    Get interview-ready

    Answer evaluation design questions with a method, and show real artifacts.

    Lessons0/1 done

    • ○Evaluation Interviews & Case PracticeCourse: AI Evaluation Engineer

    Before you move on: Answer the legal-contract summarizer question aloud in ten minutes, and fix your weakest part.