O&A Consulting · Answer

How do you evaluate a RAG system?

Q: How do you evaluate a RAG system?

Evaluate a RAG system in two halves, because the failures have different causes. For retrieval, ask whether the passages that came back actually contain what the question needs — a fluent answer built on the wrong passages is a retrieval failure wearing a generation costume. For generation, score groundedness against the passages the system actually retrieved rather than against a reference answer, and treat omission as a first-class failure, since leaving out something the sources required is the error users notice most and reference-matching catches least. Most RAG problems trace back further than either, to chunking: if the chunks don't respect meaning, retrieval cannot recover and no amount of prompt work will fix it.

Evaluate a RAG system in two halves, because the failures have different causes. For retrieval, ask whether the passages that came back actually contain what the question needs — a fluent answer built on the wrong passages is a retrieval failure wearing a generation costume. For generation, score groundedness against the passages the system actually retrieved rather than a reference answer, and treat omission as a first-class failure. Most RAG problems trace back further than either, to chunking: if the chunks don't respect meaning, retrieval cannot recover.

RAGRetrievalGroundednessChunking

01 Answer

Split retrieval from generation before you measure anything

A single end-to-end score tells you the answer was bad and nothing about why. Separate the halves and each failure points at a fix:

Retrieval: did the passages needed to answer this actually come back? If they didn't, the generator was never going to succeed, and tuning the prompt is wasted effort.
Generation: given those passages, is every claim in the answer supported by them, and does the answer cover what they required?

The common mistake is reading a confident, well-written wrong answer as a model problem. Usually the model did fine with what it was handed.

02 Answer

Ground against what was retrieved

Checking output against a stored ideal answer inherits every problem golden datasets have: the key goes stale, it encodes one person's phrasing as truth, and it punishes a better answer that doesn't match.

Checking each claim against the passages the system actually used avoids all of that, and it produces something more useful — a per-claim verdict with the supporting quote attached. That is auditable by a human who doesn't trust your evaluator, which is the person who matters.

03 Answer

Omission is the failure that hides

Hallucination gets the attention because it's visible. Omission is more common and more dangerous in high-stakes work: the answer is entirely accurate and leaves out the disclosure, the exception, or the one contradicting source that changes the decision.

Reference-matching almost never catches this, because the omitted material usually isn't in the reference either. Scoring completeness against the retrieved sources does.

04 Answer

Most of it is decided at chunking

Before retrieval or generation can be evaluated, the corpus has been cut into pieces, and that cut determines what is findable at all. A chunk that splits a thought in half retrieves as two weak matches instead of one strong one. Boilerplate and navigation text cluster with each other and pollute every neighbourhood they touch.

This is measurable rather than a matter of taste — cluster cohesion and separation tell you whether your chunks carry meaning. It's worth checking first, because no downstream metric can recover from bad boundaries.

05 Further

Where this is argued properly

The short answer above is ours, and we've written at length about how we got to it:

“Amen.” is not a chunk — chunking, measured ↗Goldens aren't ground truth ↗Aurum — grounded RAG with checked citations →