Define AI evaluations around the task your team needs done
Create representative test cases, meaningful failure categories and release checks for AI-powered business workflows.
An AI evaluation should test whether the workflow performs its intended task under realistic conditions. A few impressive examples are useful for exploration but weak evidence for release. Build a repeatable set of inputs and expected behaviours that the business owner can inspect, including cases where the correct response is to stop or ask for clarification.
Build examples around decisions and failures
Collect representative inputs using information the business is permitted to test. Cover normal requests, missing details, contradictory records and unsupported cases. Label the expected output or acceptable behaviour before looking at the model’s answer. If reviewers disagree, resolve the policy question or record the ambiguity instead of forcing a misleading single correct label.
Keep a development set for improving prompts and a separate evaluation set for checking changes. Include cases that test privacy boundaries and tool permissions when the workflow can access data or take actions. The evaluation should exercise the connected process, not just the text generated in isolation from its downstream effects.
Measure the errors that matter
Choose metrics that map to business consequences: incorrect fields, unsupported statements, missed escalations, unnecessary review or duplicated actions. Separate serious errors from harmless wording differences. Report results by important case type so strong performance on frequent easy examples cannot hide a failure in a smaller but consequential category.
Use automatic checks for structured fields, required states and exact business rules. Human review remains useful for meaning and context. If another model scores responses, validate that judge against informed reviewers and examine disagreements; Google’s judge-model guidance explicitly treats the judge itself as something to evaluate, rather than an unquestionable authority.
Turn the evaluation into a release decision
Record the model, prompt, retrieval settings and workflow version with the results. Compare changes against the same cases and investigate regressions before release. Decide which failures block deployment and which limitations are acceptable for a bounded pilot. An overall score should support that decision, not replace the owner’s understanding of the remaining risks.
Add new failure examples from operations after reviewing their cause, and maintain a separate record of unresolved issues. Retest when models, documents, rules or integrations change. A passing evaluation describes performance on the tested conditions; it does not establish that every future input will be handled correctly or that monitoring is no longer necessary.
Practical checklist
- Label normal, ambiguous and unsupported examples.
- Measure consequential errors by case type.
- Validate automated judges against informed reviewers.
- Record versions and repeat checks before releases.
Illustrative setup: testing a reply assistant
A reply assistant is tested on routine questions, conflicting delivery information and a message requesting an unauthorized discount. The evaluation checks factual support, missing-information handling and whether a draft remains unsent. A polished answer that invents a delivery promise fails even if its tone is excellent.
Common questions
How large should the evaluation set be?
Use enough examples to represent the workflow’s important cases and observed variability. There is no universal sample size that guarantees safety; expand coverage when errors or new inputs reveal gaps.
Can user satisfaction be the only metric?
No. Users may like an answer that is inaccurate or unsupported. Combine usability feedback with checks for the task’s factual, operational and permission requirements.
Further reading
Start with your actual workflow.
Turn the useful parts of this guide into a focused project brief.
Shape your project