Evaluation, Observability & Production Questions

In plain words: AI answers vary, and many can be right. So you test in layers: simple checks, real tasks, safety tests and human review.

Full answer · Outputs aren't deterministic and several answers can be acceptable, and failures depend on wording, context, model version and retrieved evidence. So you need layers — deterministic tests, retrieval tests, behavior tests, end-to-end tasks, safety tests and human review — plus a versioned golden dataset that includes ambiguity, missing data, adversarial inputs and permission boundaries, run before every change to model, prompt, retrieval or tools.

In plain words: Give it a clear checklist, make it explain before scoring, show it the evidence and compare against human scores.

Full answer · Write a rubric with observable, separately scored criteria; require reasoning before the score; give the judge the trace and evidence so it checks facts rather than plausibility; use a different model from the one under test to reduce self-preference; calibrate against human labels and track agreement; control for length and position bias with pairwise comparisons; and combine with deterministic checks. Re-check calibration periodically.

In plain words: Track whether the right text is found, whether the answer matches the sources, speed, cost per success and complaints.

Full answer · Retrieval: recall@k and MRR on a labelled set. Generation: groundedness, completeness, correct refusal rate. User-level: task success, escalation and thumbs-down rate. Operational: p50/p95 latency, cost per successful task, token usage by tenant and feature, tool and model error rates, and guardrail triggers. Tie them to business outcomes and set release thresholds, with zero tolerance for security and policy violations.

In plain words: Test the old and new versions side by side. Try the new one quietly on real traffic. Then release it to a few users, with an easy way back.

Full answer · Run the full eval suite on old and new configurations with multiple trials, compare quality, safety, latency and cost per successful task against explicit thresholds, then shadow-test on live traffic and canary a small percentage with automatic rollback triggers. Review disagreement cases by hand. A change ships only if it meets thresholds, and rollback is a rehearsed configuration switch.

In plain words: Record each step with an ID that follows the request: model calls, tools, time, cost and errors. Hide private data.

Full answer · Trace the full graph with correlation IDs from request through agent run, model calls, retrievals and tool calls: prompt/response metadata or redacted content, tool arguments and results, latency, tokens, cost, retries, errors, guardrail verdicts, versions and termination reason. Dashboards for success rate, latency percentiles, tool error rate, steps per task and cost; alerts for error spikes, loops and spend velocity; and redaction plus access control on the trace store.

In plain words: Retry only short problems, add time limits, plan a backup, save progress and make actions safe to repeat.

Full answer · Classify failures — transient, permanent, policy, quality — and treat each differently: bounded retries with backoff and jitter for transient, no retries for permanent or policy, repair-then-escalate for quality. Add timeouts everywhere, circuit breakers and tested fallbacks, idempotency for side effects, checkpointing for long runs, graceful degradation with honest partial results and a kill switch. Pin model versions and rehearse failure through fault injection.

In plain words: Measure the cost of each successful task. Reuse repeated text. Use small models for easy work. Set limits per run and per customer.

Full answer · Measure cost per successful task and attribute usage by tenant, feature and agent. Reduce with prompt caching of stable prefixes, shorter and better-targeted context, better retrieval, routing easy tasks to smaller models, batching and caching, tool-result trimming and step limits. Enforce per-run budgets, tenant quotas and a global spend circuit-breaker, and verify with the eval set that cheaper configurations still meet quality thresholds.

In plain words: Find the slow step. Run independent parts at the same time. Reuse repeated work and shorten prompts. Move long jobs to the background.

Full answer · Break it down: network, queueing, inference, retrieval, tool calls, orchestration. Parallelize independent calls, cache repeated work, shorten prompts, use smaller or faster models where evals allow, stream for perceived latency, avoid sequential review stages that add little, and move long tasks to asynchronous jobs with progress updates. Define a latency budget per workflow and watch p95 and p99.

In plain words: Look for loops, huge results, retries or a recent change. Switch it off if needed, fix the cause and add a test and an alert.

Full answer · Check spend velocity and compare traces from before and after: tokens and steps per run, tool results sizes, retries, cache hit rate, model routing and any release or provider change. Common causes are a loop, oversized tool outputs, a lost cache prefix, a prompt change or a retry storm. Contain with budgets and the circuit-breaker or a kill switch, fix the root cause through the release pipeline and add a regression test and an alert.

In plain words: Permissions and customer separation, safety limits, tests, monitoring, an off switch, a rollback plan and a named owner.

Full answer · Identity, authorization and tenant isolation verified; injection defenses and least-privilege tools; schema and business validation; rate limits, budgets, timeouts, retries and cancellation; audit logs, tracing, dashboards and alerts; an eval baseline with regression tests and version tracking; kill switch and tested rollback; incident runbook, data retention and a named owner. A demo proves the happy path; readiness proves the unsafe, expensive and adversarial paths are controlled.