Most RAG demos look brilliant on a laptop. Feed a script a twenty-page PDF, ask a question, watch it cite the right paragraph. But shipping that same pipeline into production is where the romance ends. Legal documents get sliced in half at the sentence level. Thick API references drown the important signal in boilerplate noise. Latency balloons. Users wait, twitch, and leave. We hit that wall hard. So we tore the retrieval layer down to the studs and rebuilt it as a measured, tunable system. The result was a pipeline that hits 95% recall without turning the user experience into a slideshow.

Why Demo RAG Dies in Production

The standard stack is surprisingly uniform across hobby projects and early-stage products: fixed token chunks, off-the-shelf embeddings, and a single vector search call. That simplicity is seductive, and it works when your corpus is clean, small, and syntactically predictable. Production data is none of those things. A fixed chunk of 512 tokens will happily cut through the middle of an indemnification clause in a SaaS contract. Suddenly your retrieval layer is feeding the language model half of a legal obligation and asking it to answer a liability question. The model hallucinates because the context is broken.

Large technical documents compound the issue. API documentation is full of function signatures, tables, and code blocks. A fixed window might capture the middle of a TypeScript interface but miss the function name above it and the usage example below it. The embedding vector ends up representing syntax fragments and inline noise instead of the actual capability the user is asking about. Garbage in, hallucination out.

Chunk by Structure, Not by Token Count

The first change we made was to stop thinking about chunks as bags of tokens. Chunks are semantic units. The right strategy depends entirely on what you are indexing.

For legal documents, we moved to recursive chunking that respects the document hierarchy. It treats sections, subsections, and clauses as boundaries. A clause stays intact because a clause is a unit of meaning. If you slice through it, the legal logic leaks out.

For API docs, structure-aware chunking treats functions, classes, and endpoints as atomic. One chunk might contain a function signature, its arguments, and its docstring. It does not spill arbitrarily into the next utility function just because a token counter clicked over. This keeps the embedding focused on a discrete capability.

Support tickets are messier. They are conversational, threaded, and nonlinear. Fixed chunks will grab a status update from an engineer and a customer complaint from the same thread and pretend they form a coherent unit. We switched to semantic chunking, splitting when the topic or speaker shifts rather than when the token budget runs out.

Internal wikis are often the sloppiest data in an organization. Formatting is inconsistent, headers are missing, and sections run together. For these, we use LLM-based chunking. A small model reads ahead and identifies logical boundaries before we ever generate an embedding. It costs more upfront than a character split, but the retrieval quality pays for itself immediately.

Hybrid Retrieval: Combine Signals

Vector search is powerful but it has blind spots. Paste in an exact error code like ERR_CONNECTION_REFUSED_0x800 and similarity search can return a troubleshooting guide for an unrelated module because the embedding space clustered them close together. Exact matches matter, and vector search alone can smooth them away.

Keyword search with BM25 solves the exact-match problem beautifully. But it chokes on conceptual distance. If a user asks about "performance degradation under heavy load," BM25 will miss a diagnostic note that describes "slow throughput during traffic spikes" because there is not enough keyword overlap.

We stopped choosing sides and started running both in parallel. Vector and keyword searches each return their own ranked lists. We merge them with Reciprocal Rank Fusion. RRF is simple and brutal in its effectiveness. It scores each document based on where it sits in each list. Documents that sit near the top in both systems get a massive boost. Documents that only one engine loves still earn a place in the final candidate set.

After fusion, we run the top candidates through a cross-encoder reranker. This is not free. It adds roughly 50 milliseconds of compute. It also increases recall by 15%. The cross-encoder evaluates the full query and each candidate chunk together, producing a relevance score far more nuanced than a bi-encoder embedding ever could. That extra 50 milliseconds is a bargain. It prevents you from shipping a garbage context window to the LLM and spending two seconds waiting for a confused or hallucinated answer.

Users do not write queries like search engineers. They type "app broken." They paste cryptic log fragments. They ask vague, ambiguous questions. If you send those raw strings straight to the index, you get garbage back.

We transform every query before it touches the retrieval engine.

First, query expansion. The system generates multiple search terms from a single short question. A user asks, "How do I fix the timeout?" The engine expands that to cover connection timeouts, read timeouts, gateway timeouts, and retry logic. This approach alone moved our recall from 78% to 96%.

Second, query decomposition. Complex questions get broken into smaller sub-questions. A query like "What's the refund policy for enterprise customers past 90 days and how does it differ from monthly plans?" becomes two focused searches rather than one bloated embedding lookup. Each sub-question hits the index independently. The results are stitched back together downstream. This keeps retrieval narrow and precise, which stops the dilution that happens when a single embedding tries to match a dozen concepts at once.

Let Bayesian Search Tune Your Pipeline

If you are still hand-tuning chunk size, overlap ratios, and retrieval weights, you are leaving performance on the table. We stopped guessing.

We defined a search space where chunk size, overlap percentage, vector-versus-BM25 weights, and reranker thresholds are all variables. Then we applied Bayesian optimization. Instead of grid-searching through hundreds of random configurations, Bayesian search builds a probabilistic model of what works. It proposes a configuration, observes the recall and latency, updates its beliefs, and proposes the next one. Over time it converges on balances a human would never stumble into manually.

It found combinations we never would have tried. Smaller chunks with heavier overlap. A slightly lower weight on dense vector search paired with a more aggressive reranker threshold. These non-obvious tradeoffs gave us both higher recall and lower latency.

This is not a one-time setup task. We re-run hyperparameter optimization monthly. Your corpus drifts. User behavior shifts. Your pipeline should adapt instead of rusting in place.

The Payoff

The raw output of that rebuild is hard to argue with.

Recall at position ten went from 78% to 95%. When the correct answer lives in our knowledge base, we surface it nineteen times out of twenty. Latency at the 95th percentile fell from 850 milliseconds to 320 milliseconds. The chat feels instant instead of ponderous.

Better retrieval gave the language model better grounding. Hallucination rate dropped from 12% to 3%. When the model has the right context in front of it, it stops inventing facts. Cost per query fell by 38%. Faster, sharper retrieval means fewer tokens wasted on irrelevant context, retry loops, and verbose but useless prompts.

Build It Like Infrastructure

If you are moving from prototype to production, treat retrieval as infrastructure code rather than a configuration