I run a digital twin on my website. It answers questions about my life and skills. I gave it one strict rule: never make things up. If someone asks about a skill I do not have, it must admit it does not know. For months, I believed the system was working. I tested it by hand here and there, and the answers looked solid. Then I built a proper evaluation harness. The numbers hit hard. Out of 35 questions, nine contained outright lies. Out of eight questions designed to be unanswerable, the model refused only four. My anti-hallucination prompt failed roughly a quarter of the time. I was shipping a product that lied to its users.
A Dead-Simple Retrieval Setup
I did not spin up Pinecone or any heavyweight vector database. The entire setup sits on a plain JSON file. My code splits my profile into discrete sections. When a question arrives, the system calculates cosine similarity between the query and each chunk of text, selects the closest matches, and stuffs them into the prompt as context. The model then generates an answer based strictly on what it sees in that window.
For a small personal site serving a narrow set of facts, this approach is fast and costs next to nothing. There is no network round-trip to a remote vector store, no indexing overhead, and no complex orchestration. You read the file, score the chunks, build the prompt, and go. But simplicity on the backend does not guarantee honesty in the output. A lightweight pipeline can still produce serious problems when the model decides to improvise. The gap between "here is the context" and "here is what I will say about it" is where hallucinations live. You can hand the model a paragraph about your work history and still get back a confident fabrication about a programming language you have never touched.
The Numbers That Broke My Confidence
For months, I treated manual spot-checking as sufficient coverage. I would open the chat, ask a question I already knew the answer to, and nod when the response looked right. That was my testing strategy. It felt thorough because I was using the interface myself. It was not.
When I finally wrote an evaluation harness that could run systematically, the picture changed. The test suite fired 35 questions at the twin. Nine answers contained lies. I also included eight questions that had no answer anywhere in my profile. The model should have declined them all. It refused only four. My carefully crafted anti-hallucination prompt, the one that included absolute language about never making things up, failed about 25 percent of the time. One in four. That is not a rounding error. That is a broken product.
Stop Testing With Friendly Questions
You cannot find bugs by simply using your own product. You find them by trying to break it. My manual tests were too friendly. I only asked questions where I knew the exact answer, which meant I was subconsciously guiding the model toward safe territory. I never probed the edges. I never asked about skills I wished I had, or about experiences that never happened.
Real testing requires adversarial intent. You have to craft questions designed to make the AI fail. You want it to slip up in the lab so it does not slip up in front of a visitor. A test suite that only confirms what you already believe is just a dressed-up demo. If you are not actively manufacturing edge cases and trap questions, you are not testing. You are hoping.
The Two Mistakes That Mattered
Prompting is not a guarantee. A long, detailed instruction telling an AI not to hallucinate is just a suggestion dressed up as a command. The model may follow it most of the time, but it will ignore the instruction the moment statistical pressure pushes it elsewhere. Temperature, token probability, and the shape of the training data all weigh heavier than a sentence in your system prompt. You must measure obedience with data, not hope. A strong instruction is not a verified fact. It is a request, and requests get denied. If your entire safety strategy rests on wording your prompt firmly, you have built a guardrail out of tissue paper. You need an eval harness that counts how often the model obeys, under what conditions, and why it fails when it fails. Numbers do not care about your tone of voice.
The evaluation loop was flawed. Here is a subtle trap that almost got me. My original testing tool ran the retrieval process twice. The first run fetched context to check against the ground truth. The second run fetched context for the actual answer generation. In practice, this meant the chunks the judge saw could differ from the chunks the model saw. The judge was grading the answer against data the AI might never have received. An evaluation that grades the wrong input is worse than no evaluation. It gives you a false sense of security. You look at the score, see a high pass rate, and relax. Meanwhile your users are