𝗜 𝗔𝗱𝗱𝗲𝗱 𝗮 𝗩𝗲𝗿𝗶𝗳𝘆 𝗟𝗮𝘆𝗲𝗿 𝘁𝗼 𝗠𝘆 𝗟𝗼𝗰𝗮𝗹 𝗥𝗔𝗚 𝘁𝗼 𝗖𝗮𝘁𝗰𝗵 𝗛𝗮𝗹𝗹𝘂𝗰𝗶𝗻𝗮𝘁𝗶𝗼𝗻𝘀

I build a local research assistant using Ollama. It runs on my own papers. Nothing leaves my machine.

I wanted to stop hallucinations. A tool that cites a wrong number while sounding sure is dangerous.

I added a verification layer. It works in three steps:

  • Break the answer into small claims.
  • Use an LLM to check each claim against the source.
  • Flag claims the source does not support.

The results taught me a hard lesson. I was wrong about my own data twice.

First, the model gave a real number but used the wrong context. It cited an AUROC of 0.804 for a test set that does not exist. The number was real. The context was a lie. My verifier passed it because the digits matched.

Second, the model grabbed a number from a different part of the paper. It attributed a value to the wrong experiment.

Here is what I learned from testing this:

  1. Verification only catches absent values. If a number is not in the text at all, the verifier catches it. If the number is real but attached to the wrong fact, it often fails.

  2. Same-model judges have blind spots. If the same model writes the answer and judges the answer, it rubber-stamps its own mistakes. Using a different model to judge helps catch misattributed numbers.

  3. A flag does not always mean a lie. A flag can mean three things:

  • A real hallucination.
  • A retrieval error where the source was not found.
  • A true fact that was not in the retrieved text. When you see a flag, try re-retrieving the data instead of just deleting the claim.
  1. You need ground truth. You cannot measure hallucinations without knowing the real answers. I almost published two false findings about my own work. A simple search through my files fixed both errors.

Practical advice for your RAG:

  • Use a different model to judge than the one that answers.
  • Focus on better retrieval. Most "hallucinations" are just retrieval failures.
  • Treat flags as a prompt to look deeper, not just a sign of error.

Source: https://dev.to/sysoft/i-added-a-verify-layer-to-my-local-rag-to-catch-hallucinations-it-caught-me-being-wrong-twice-1jm

Optional learning community: https://t.me/GyaanSetuAi