𝗦𝘁𝗼𝗽 𝗬𝗼𝘂𝗿 𝗔𝗜 𝗕𝗼𝘁 𝗙𝗿𝗼𝗺 𝗟𝘆𝗶𝗻𝗴

𝗦𝘁𝗼𝗽 𝗬𝗼𝘂𝗿 𝗔𝗜 𝗕𝗼𝘁 𝗙𝗿𝗼𝗺 𝗟𝘆𝗶𝗻𝗴

I built a customer support bot. It lied to users. It made up features and prices. This kills trust.

I tried a few fixes first.

• Better prompts. The bot still lied.
• Simple FAQ lists. These were hard to maintain.
• Dumping all data into the prompt. This caused more errors.
• Lowering temperature. The bot sounded robotic.

The problem is clear. LLMs generate text. They do not retrieve facts.

I switched to Retrieval-Augmented Generation. Now the bot only sees relevant data.

Here is the new flow:

• Break docs into small pieces.
• Turn text into embeddings.
• Store them in a vector index.
• Search for the best match during a user query.

I added a similarity gate. If the match score is high, the bot answers. If the score is low, the bot says it does not know. It then connects the user to a human.

Three lessons from this project:

• Tune your threshold. Find the balance between truth and silence.
• Use overlapping chunks. This keeps the context.
• Use a human fallback. This prevents the bot from becoming a liability.

How do you stop AI hallucinations? Share your setup below.

Source: https://dev.to/__c1b9e06dc90a7e0a676b/my-support-bot-kept-making-stuff-up-heres-how-i-fixed-it-31ij