Estimated reading time : 2 min · Published October 1, 2026
Define a task and its required quality
Choose a specific task: suggest headings, classify enquiries, translate a page or answer from documentation. Define the expected result and what would constitute a significant error. Internal writing assistance has different consequences from a public answer about price, timing or contractual terms.
The NIST AI Risk Management Framework and its generative-AI profile offer references for organising assessment. They do not approve a particular product. Your criteria should connect risks to the task, audience and use of its output.
- Task and audience.
- Verifiable outcome.
- Acceptable and blocking errors.
- Decision owner.
Review data and the exit path
List data sent to the service, people who can access it, retention rules and stated usage terms. Verify these in the supplier’s documentation and contract. Do not assume every product from one company has the same conditions. A consumer account, API and enterprise plan may differ.
Prepare anonymised or clearly labelled synthetic examples for initial trials. Define what can be exported, how to change suppliers and which task remains possible during an outage. Store instructions, sources and evaluations so the next project team can understand them.
- Data permitted for testing.
- Terms specific to the offer.
- Access, retention and export.
- Manual fallback.
Build a small but demanding test set
Collect representative and difficult cases: missing information, conflicting sources, ambiguity, another language and attempts to divert instructions. Define reference answers or acceptance criteria before testing. A tool that declines to answer when information is missing may be more useful than one that produces a convincing invention.
Measure accuracy, omissions, references, latency and cost separately in the chosen scenario. Review multiple outputs because generation may vary. Retain the model or offer version when known, useful settings, test date and examples. A single total score should not conceal failure on an essential task.
- Representative and edge cases.
- Criteria agreed before testing.
- Errors documented per example.
- Test version and date.
Limit deployment and reassess over time
Begin with reviewed outputs. For a public assistant, restrict possible actions, make sources available and provide a human contact when a request falls outside its remit. Test recovery after outages, invalid answers and document changes. Observe outcomes without needlessly retaining personal data.
Repeat the tests when models, instructions, documentation or capabilities change. The source-verification guide supports editorial review. The AI resource directory offers official starting points for comparing services without guaranteeing their results.