Measuring the Wrong Half of Your RAG Pipeline

Your RAG eval dashboard is green. Faithfulness is 0.91, answer relevance is 0.88, and the LLM-as-judge harness you spent two sprints building says the system is doing fine. Meanwhile, a user just asked a question whose answer sits in a document your retriever never surfaced, and your model wrote a confident, well-structured, entirely useless response about something adjacent. The judge scored it highly. It read well. It was grounded in the passages it did get. It just answered the wrong question with material that had nothing to do with what the user needed.

This is the quiet structural flaw in how most teams evaluate retrieval-augmented generation: they grade the essay and never check whether the student was handed the right book. A RAG system is two machines bolted together — a retriever that decides what the model gets to see, and a generator that decides what to do with it. Almost every eval harness in production measures only the second machine. The first one, the one that actually determines the ceiling on answer quality, runs unmonitored.

The consequence is not just a blind spot. It's that a single aggregate score actively hides the failure. When retrieval degrades — because you swapped an embedding model, re-chunked your corpus, or your index quietly went stale — the generation metric absorbs the damage. The model keeps producing fluent, faithful-to-its-context answers. Faithfulness stays flat. Answer relevance dips a point or two, well within noise. Nobody pages. And your retrieval recall has fallen from 0.9 to 0.7 while every instrument on your dashboard reads normal.

Why One Score Makes Both Halves Un-Debuggable

The seductive thing about an end-to-end score is that it feels like the honest metric. The user experiences the final answer, so why not measure the final answer? Because a composite number tells you that something is wrong without ever telling you where, and in a two-stage pipeline "where" is the entire question.

Consider the four things that can happen on any given query:

• Good retrieval, good generation — the answer is right for the right reason.
• Good retrieval, bad generation — the answer-bearing chunk was in the context and the model still fumbled it, hallucinated, or ignored it.
• Bad retrieval, bad generation — the model never had a chance; garbage in, garbage out.
• Bad retrieval, "good" generation — the model wrote a clean, faithful answer grounded in the wrong documents. This is the dangerous one, because it looks like success to every downstream metric.

An end-to-end faithfulness score cannot distinguish these cases. Worse, faithfulness specifically rewards the fourth one. Faithfulness asks: is the answer supported by the retrieved context? If retrieval brought back the wrong passage and the model dutifully summarized it, the answer is perfectly faithful — to the wrong source. You have built a metric that gives its highest marks to a confident lie, as long as the lie is internally consistent with the mistake that produced it.

So when your single number ticks down, you're stuck. Is the retriever missing documents? Is the reranker mis-ordering them? Is the chunk size wrong? Is the model ignoring context it was given? Is the prompt bad? You can't tell, because you measured the sum and now you're trying to reverse-engineer the addends. You end up A/B testing prompt tweaks against a problem that lives entirely in the vector index — flying with one instrument, adjusting the throttle to fix a navigation error.

Instrument Retrieval On Its Own Terms

The fix is to stop treating retrieval as an invisible upstream dependency and start scoring it as a first-class system with its own metrics, its own labeled data, and its own dashboard. Retrieval is, at bottom, an information-retrieval problem — the same one search engines have measured rigorously for decades — and the metrics already exist. You just have to actually compute them.

The foundation is a labeled set: a collection of representative queries, each paired with the document or chunk that actually contains the answer. This is the unglamorous, expensive part, and it's the part teams skip. But without ground truth you cannot say whether retrieval succeeded — you can only guess. A few hundred well-chosen query-to-gold-chunk pairs is enough to start, and it's the single highest-leverage artifact in the whole eval stack. Once you have it, the metrics come almost for free:

• Recall@k — of all the relevant chunks, what fraction showed up in the top k you retrieved? This is the one that matters most, because it answers the only question that determines the ceiling: was the answer even in the context window the model saw? If recall@k is 0.7, then 30% of the time your generator is being asked to answer from material that doesn't contain the answer. No prompt engineering fixes that.
• Precision@k — of what you retrieved, how much was actually relevant? Low precision means you're stuffing the context with noise, burning tokens and diluting the signal the model has to find.
• MRR (Mean Reciprocal Rank) — how high up did the first relevant result land? This matters because position isn't neutral; models attend more reliably to material near the top of the context.
• nDCG@k — a rank-aware score that rewards putting the most relevant chunks highest. This is the metric that actually reflects whether your reranker is earning its keep.

Loading…