Production-Readiness Review

This final lesson turns the whole path into one tool you can use before every launch: a production-readiness review. It is a checklist across the eight pillars with a simple scoring rule. Run it honestly. The point is not a high score. The point is to find the gap that would otherwise find you in production.

How to run the review

• Do it as a conversation with the builder, an ops or on-call person, and someone from security or risk. Do not fill in a form alone.
• For each item, ask for evidence, such as a link to the dashboard, the runbook or the eval run. Do not accept assurances.
• Mark each item Pass, Partial or Fail. Any Fail on a blocker item, marked ★, stops the launch.
• Record an owner and a date for every gap, and review again after the fixes.

1. Observability

• ★ Every run produces a trace with model calls, tool calls, versions, tokens, delay and cost.
• Traces have secrets and unneeded personal data removed. Retention and access are defined.
• Dashboards show success, delay, errors, steps per run, tool errors and cost per run.
• Alerts exist for error spikes, the speed of spending and loops, and each one links to a runbook.

2. Evaluation

• ★ A regression set exists with at least 50 cases, including edge cases and hostile ones, and it runs several tries per case.
• Outcomes (the end state), not just text, are graded. Judges are checked against human labels.
• ★ An eval gate blocks releases that make quality, safety or cost worse.
• Production failures flow back into the dataset. Sampling of live traffic tracks drift.

3. Guardrails and security

• ★ The lethal-trifecta check is done. No agent has private data, untrusted content and a way out without an extra control.
• ★ Tool permissions are enforced in code, with least privilege and short-lived, scoped credentials.
• Code running and browsing are sandboxed, and the way output is shown cannot leak data.
• A suite of hostile tests runs in CI. MCP and tool dependencies are checked and pinned.

4. Human oversight

• ★ Every action is risk-tiered. Tier 2 and 3 actions need approval or tight automatic-approval rules.
• The approval screen shows the action, the evidence, the impact and the confidence. Reviewers can edit and reject.
• Escalation triggers are defined. Hand-offs to people keep the context.
• An append-only audit log records who or what authorized each important action.

5. Cost and usage

• ★ Budgets for each run (tokens, steps and dollars) and a global spending circuit breaker are active.
• Usage events assign spend to customer, feature, agent, model and run.
• Unit economics, such as cost per successful task and margin, are tracked. Quotas degrade gently.
• Provider rate limits and capacity are monitored.

6. Reliability

• ★ Timeouts, limited retries with jitter and idempotency keys exist on every outside call.
• Fallback models and paths have been tested with evals. Circuit breakers and graceful reduction exist.
• Long-running work is queued and checkpointed. Releases drain safely.
• Model snapshots are pinned, and retirement dates are tracked. Fault injection has been run.

7. Change management

• ★ Releases are fixed bundles of prompts, model, tools, search and policy, with an ID on every trace.
• The pipeline includes review, an eval gate, shadow or canary runs, and a step-by-step ramp.
• ★ A kill switch and a rehearsed rollback exist.
• A changelog records what changed, why and the eval results.

8. Governance

• ★ The agent is in the register with named business and technical owners and a risk class.
• Data flows, vendors, retention and deletion are written down. Data processing agreements are in place where needed.
• Users are told that they are dealing with AI and how to reach a person.
• ★ An incident runbook exists, and a tabletop exercise has been run.

Scoring and a 30, 60, 90 day plan

Count Pass as 2, Partial as 1 and Fail as 0, across all items. To launch, every ★ item must pass, and the overall score must be at least 75 percent. Below that, use a 30, 60, 90 day plan. In the first 30 days, close every blocker and install the kill switch. By 60 days, get the eval gate and the cost controls working in CI and in production. By 90 days, complete a red-team exercise, a full incident tabletop and a first quarterly review.