Regression Tests & Release Gates

A test set only helps if it is used on every change. Regression testing makes this automatic. A release gate is a rule that stops a change from shipping if it makes quality worse. This lesson shows how to put evaluation into the build process, set thresholds, and roll out changes safely.

What can change

Many things can change an AI system's behavior: the prompt, the model version, the documents, the tools, the code and the settings. Any of these can lower quality in a way that nobody notices by eye. So treat all of them as code. Put them in version control. Test every change.

Regression testing

A regression is when something that worked before stops working. Your test set is a set of things that must keep working. Run it on every change. Compare the new scores with the last good version. Look at case-level differences, not only the average.

RELEASE GATE for {{feature}}

RUN: full test set + must-never-happen set + cost and speed checks

BLOCK THE RELEASE IF
  - Any must-never-happen case fails
  - Overall pass rate drops by more than {{agreed amount}} compared with the current version
  - Any important group (slice) drops by more than {{agreed amount}}
  - Average cost per task rises by more than {{agreed amount}}
  - 95th-percentile response time rises above {{agreed limit}}

REVIEW BY A PERSON IF
  - More than {{n}} individual cases changed from pass to fail, even if the average is fine

RECORD: version, date, scores, who approved, link to the report

Set thresholds with care

Test scores are noisy, especially with small sets. A drop of one case may be chance. Decide thresholds with the team, based on how much a drop would hurt. Be strict on must-never-happen cases, and allow a small margin elsewhere. Review thresholds when the set grows.

Put it in the pipeline

• Run a fast check set on every code or prompt change.
• Run the full set before a release, or each night.
• Block merges that fail the gate, and show the failing cases in the report.
• Store results with the version, so you can chart quality over time.

Roll out in steps

Even good tests miss things. Roll out gradually, so real traffic shows problems while few people are affected.

• Shadow: the new version runs on live traffic, but users still see the old one. Compare results.
• Canary: send a small share of traffic to the new version. Watch quality and cost.
• A/B test: split traffic between two versions to measure which does better on a business measure.
• Rollback: keep the old version ready, and switch back quickly if a problem appears.

Model and provider changes

Providers update and retire models. Behavior can shift even when you change nothing. Pin the model version where you can. Re-run the full test set when you move to a new version. Never switch the model on a Friday afternoon without a rollback plan.