Search
Enter at least two characters without personal information.

    LibraryArticles & FAQ Your projectRequest a quote
    LanguageEN
    Practical guides

    Evaluate an AI tool before integrating it into a website

    A successful demonstration does not prove that an AI tool suits your visitors or team. Selection requires test cases, explicit limits and a way to resume the task.

    Go to the method

    Person reviewing documents on a screen
    AI-generated illustration

    Define a task and its required quality

    Choose a specific task: suggest headings, classify enquiries, translate a page or answer from documentation. Define the expected result and what would constitute a significant error. Internal writing assistance has different consequences from a public answer about price, timing or contractual terms.

    The NIST AI Risk Management Framework and its generative-AI profile offer references for organising assessment. They do not approve a particular product. Your criteria should connect risks to the task, audience and use of its output.

    • Task and audience.
    • Verifiable outcome.
    • Acceptable and blocking errors.
    • Decision owner.

    Review data and the exit path

    List data sent to the service, people who can access it, retention rules and stated usage terms. Verify these in the supplier’s documentation and contract. Do not assume every product from one company has the same conditions. A consumer account, API and enterprise plan may differ.

    Prepare anonymised or clearly labelled synthetic examples for initial trials. Define what can be exported, how to change suppliers and which task remains possible during an outage. Store instructions, sources and evaluations so the next project team can understand them.

    • Data permitted for testing.
    • Terms specific to the offer.
    • Access, retention and export.
    • Manual fallback.

    Build a small but demanding test set

    Fictional classification of 100 cases: 6 true positives, 2 false positives, 4 false negatives and 88 true negatives.
    Illustration · Fictional exercise, with no performance attributed to a tool. REF is the reviewed reference and MODEL the tested decision; + means “needs review”. TP: true positive; FP: false positive; FN: false negative; TN: true negative.

    Collect representative and difficult cases: missing information, conflicting sources, ambiguity, another language and attempts to divert instructions. Define reference answers or acceptance criteria before testing. A tool that declines to answer when information is missing may be more useful than one that produces a convincing invention.

    Measure accuracy, omissions, references, latency and cost separately in the chosen scenario. Review multiple outputs because generation may vary. Retain the model or offer version when known, useful settings, test date and examples. A single total score should not conceal failure on an essential task.

    Fictional request-classification exercise, with no result attributed to a tool: of 100 reference cases, 10 require human review and 90 do not. The rule flags 8: 6 genuinely require review (TP) and 2 are false alarms (FP). It misses 4 review cases (FN) and correctly excludes the other 88 (TN). Accuracy is (6 + 88) / 100 = 94 %. Yet precision of the flags is 6 / 8 = 75 %, and recall is only 6 / 10 = 60 %.

    These metrics answer different questions: how many decisions are correct, how many flags are justified and how many target cases are found. Google’s classification course explains their denominators; high accuracy can hide missed rare cases. Fix the positive-case definition, any threshold and reference set before comparing, and keep all four counts. A zero denominator makes the fraction undefined; do not arbitrarily turn it into success. These classification metrics alone measure neither factual quality of generated prose nor general reliability.

    • Representative and edge cases.
    • Criteria agreed before testing.
    • Errors documented per example.
    • Test version and date.

    Scroll the table horizontally to read every column. With a keyboard, focus the table area and use the arrow keys.

    Fictional exercise: three metrics for one classification
    MetricCalculation on the fictional setWhat the denominator represents
    Overall accuracy(6 + 88) / 100 = 94 %All classifications, correct or incorrect
    Flag precision6 / (6 + 2) = 75 %The eight flagged cases
    Recall6 / (6 + 4) = 60 %The ten cases requiring review
    Missed review cases (FN)FN = 4Four actual positive cases not flagged

    Limit deployment and reassess over time

    Begin with reviewed outputs. For a public assistant, restrict possible actions, make sources available and provide a human contact when a request falls outside its remit. Test recovery after outages, invalid answers and document changes. Observe outcomes without needlessly retaining personal data.

    Repeat the tests when models, instructions, documentation or capabilities change. The source-verification guide supports editorial review. The AI resource directory offers official starting points for comparing services without guaranteeing their results.

    Reference documents

    Further reading

    This library offers routes through tool selection, sources, confidential data and reproducible evaluation. Use it to prepare options to investigate, then apply your own protocol to the project’s tasks and constraints.

    WORLD AI GUIDE : AI tool selection and evaluation guides

    Content updated on

    Acceptance matrix to adapt to your project

    These proposed checks use synthetic cases. Decide the required behaviour with the team, record the result and assign unresolved gaps before release.

    Scroll the table horizontally to read every column. With a keyboard, focus the table area and use the arrow keys.

    Test cases, expected outcomes and useful evidence
    CaseExpected outcomeEvidence to retain
    An answer cites a source that does not support its claim.Mark it unsupported and apply the defined review rule.Sample question, exact answer, consulted source and passage supporting or contradicting the claim.

    Frequently asked questions

    How many tests are needed?

    It depends on the range and consequences of the tasks. Start with representative examples, add previously observed failures and cover critical situations. A well-designed small set is more useful than many easy demonstrations.

    Should generated content be reviewed?

    The review level depends on context. Before editorial publication, check facts, sources, rights, consistency and wording. Prices, availability and commitments must not be invented from model output.