How the test was set up

The author built a tiny retrieval-augmented generation (RAG) system around two payment-rulebook documents and ran two checks:

  • Recall test – did the correct page appear among the top five results? Result: 60 %.
  • Answer test – was the final answer produced by the generator correct? Result: 90 %.

A 40 % gap looked impossible. If the retriever missed the right page four times out of ten, how could the model still answer correctly nine times out of ten?

First attempts to “fix” the retriever

The developer tried two common tricks:

  • Hybrid search – mixing lexical and vector signals. Recall stayed the same.
  • Reranker – reordering the five fetched pages. Recall rose to 70 % but still lagged far behind the 90 % answer accuracy.

Both tools only reshuffle what has already been fetched; they cannot conjure a page that never entered the candidate set. The problem lay elsewhere.

The metric, not the model, was broken

Instead of judging success by the page label, the author inspected the actual facts in the retrieved chunks. In three out of four “misses,” the correct fact was present, but it lived on a different page than the test script expected. The evaluation penalised the retriever for finding a correct answer on an unexpected page.

When the metric switched to “does any retrieved chunk contain the required fact?” recall jumped to 90 %, matching answer accuracy. The retriever worked; the evaluation framework did not.

Why traditional recall can be misleading

  • Page-level labeling creates phantom failures. A single fact may appear on several pages. Tagging only one page as ground truth treats all other correct hits as errors.
  • Small corpora amplify the effect. With few documents, a single mis-labelled page can swing recall dramatically while answer accuracy stays steady.
  • Embedding noise hides facts. Vector scores whole pages; surrounding legal or technical text dilutes the relevance signal of the target sentence, pushing the page down the ranking even though the fact is present.

Practical takeaways for RAG practitioners

  • Separate ranking from retrieval failures. Rerankers only fix the former; if the correct chunk never makes the candidate set, reordering helps nothing.
  • Label test data at the fact level. Tie each query to the specific piece of information it needs, not to a single document identifier.
  • Don’t trust large-scale benchmarks for tiny datasets. Small, domain-specific corpora behave differently, and generic recall scores can be deceptive.
  • Watch embedding granularity. A page-sized chunk packs many words, and surrounding legal text can lower the rank of the fact you care about.

Bottom line: a high answer-accuracy score can coexist with a low traditional recall figure when the evaluation is mis-aligned with the task. Fixing the metric, not the model, saves time, cuts false alarms, and yields more trustworthy RAG deployments.