The RAG evaluation checklist we run beforeanything ships

Everyone agrees RAG systems should be evaluated. Fewer teams can say what their evaluation actually consists of. This is the checklist we run before any retrieval system answers a real question — steal it wholesale.

1. A labelled set built from real queries

Two hundred questions minimum, drawn from support tickets, search logs or the questions your team already answers. Each one needs the answer a domain expert considers correct, and — this is the part people skip — the documents that answer should come from.

Synthetic questions test whether your system handles clean input. Your users will not provide any. The gap between the two is exactly where retrieval fails.

2. Chunking decided by measurement, not intuition

Test at least three chunking strategies against the labelled set: naive fixed-size, structure-aware (headings, sections), and small-chunks-with-rich-metadata. On our last four engagements the obvious choice won once.

• Record recall@k for k = 3, 5 and 10 separately — the right k is an output, not an input.
• Keep the failing queries. They are the specification for every future improvement.
• Re-run whenever the corpus changes shape, not just when the code changes.

3. A decline threshold, calibrated

The system must be able to say it does not know. Set the similarity threshold from the labelled set: the score below which answers are more often wrong than right. Then escalate those to a human instead of generating.

A refusal costs seconds. A confident wrong answer costs a customer.

4. Citation checks on every answer

Every generated claim should trace to a retrieved passage. We grade this automatically with a second model pass, and anything ungrounded is either revised against its sources or escalated. The checker needs its own spot-checks — an unevaluated checker is just confidence with extra steps.

5. Shadow mode before launch

Run against live traffic silently for at least a week. You get real queries, zero user risk, and an honest accuracy number before anyone relies on the output. Twice now this has caught a corpus change that broke chunk boundaries days before launch.

6. Cost per resolved task, tracked from week one

Retrieval cost scales with usage in ways that are easy to miss during evaluation. Track tokens, embedding calls and reranking separately, per resolved task — not per query, since agentic patterns issue several. On our last engagement, tightening top-k and adding a cache cut this 78% with no measurable accuracy change.

7. A regression run on every change

Prompts, models, chunking, corpus updates — all of it goes through the labelled set before it ships. The run takes minutes and has caught three pre-launch regressions this year alone, including a model upgrade that was better on generic benchmarks and three points worse on the client's domain.

What "done" looks like

A number you can put in writing: accuracy against the labelled set, the decline rate, and the cost per resolution. If a vendor or a team cannot give you those three, they have changed the system rather than improved it — and you cannot tell the difference.