To evaluate a RAG, separate documentary research from the editorial staff. A badly reformulated relevant passage and a fluid response based on the wrong document are two different defects. Build a benchmark game, then examine the retrieved sources and the claims produced.
Build a realistic question set
Take frequent requests, inaccurate formulations and a few questions outside the scope. For each, write the expected answer with the passages that justify it. Have this reference validated by a person who knows the subject. Keep questions independent of the system settings to avoid optimizing only the examples already used during development.
Control what was found
Before noting the style of the response, open the recovered documents. Do they actually contain the necessary information? Is the correct version used? Was an important condition cut off when slicing? Classify errors: document absent from the corpus, insufficient search, too wide passage or obsolete version. This analysis indicates which part of the string to modify.
A small acceptance test that reveals several families of errors
The following grid describes demonstration cases, with no measured results on a client system. Each question must have a validated reference before the test. The reference can specify that you should not answer. This prevents a model from being rewarded for giving a plausible answer to a question that the corpus does not solve.
| Test question | Waiting to check | failure to classify |
|---|---|---|
| What does this offer include? | Limited response to the scope of the current document | Added service without source |
| Does this condition apply to all formulas? | Exception or difference in scope kept | general rule presented without its condition |
| What is the price of an undocumented option? | lack recognition and useful orientation | Invented amount |
| What does an old procedure withdrawn say? | Lack of restitution as current rule | Obsolete version still recovered |
| Can I consult this reserved document? | Respect of user rights | Unauthorized passage exposed in the response |
For each failure, keep Question, Retrieved Documents, Produced Answer, and Reviewer's Decision. If multiple defects coexist, write them down separately. A relevant source can be accompanied by an invented sentence; It would be misleading to summarize this case by “recovered source, therefore success”. The statement should also allow another person to understand why the answer was accepted or refused.
Evaluate the answer claim by claim
Check accuracy, completeness, fidelity to sources and recognition of uncertainty. A clickable quote is not enough if the passage quoted does not support the sentence. An automatic evaluator can help sort the results, but its judgment needs to be controlled. Hugging Face presents an evaluation approach with questions, reference answers and explicit criteria; Adapt it to the risk of your business.
Fictional example: a misquoted time limit
The document distinguishes between preparation and delivery. The wizard does not add up the steps and announces an incorrect overall delay. Research has found the right passage, but the answer remains false. The patch is about formulation and calculation control, not necessarily on the vector basis. Conversely, an outdated source first requires a documentary correction.
Use measurements without hiding important cases
Two simple measures can help the pilot: the proportion of questions for which at least one expected passage is recovered, then the proportion of answers accepted according to your grid. These are indicators proposed for this protocol, not a universal definition of recall or an official quality score for RAGs. Publish their denominator and keep the question categories separate. A good overall result can mask the failure of all unanswered cases.
- Freeze a batch of questions and a version of the corpus. Reserve another batch to verify that the settings do not work only on the examples already examined.
- Have part of the answers read again by two people. Disagreements often indicate that the grid or reference lacks precision.
- If you are using an evaluator model, confront its decisions with this proofreading. An automatic assessment does not replace proof in the document.
- Compare only one important change at a time and archive the outputs. Prioritize that no regression affects accesses or cases outside the perimeter.
The Hugging Face — RAG evaluation Presents a benchmark approach with references. Our grid completes this approach with decisions adapted to the content of a site and its users. For a public assistant, a careful and verifiable response may be preferable to a more comprehensive response that adds a missing condition. The transition to a human must be part of the authorized results.
If the defects appear to be related to the writing rather than to the documentary research, the LLM Comparison Protocol to Business Tasks Helps prepare a targeted trial. Keep the same sources and questions so as not to confuse the causes.
Compare a single change at a time
Archive Setup, Corpus, Questions and Results before each test. Then modify a parameter, then relaunch the same cases, including the expected refusals. Also note response time and operating cost. The best configuration is the one that reaches your quality criteria on useful questions. First prepare a Mastered documentary base Rather than looking for a spectacular score on a few demonstrations.