Most teams build their first retrieval system the same way: slice every document into fixed 512 token chunks, push them

Most teams build their first retrieval system the same way: slice every document into fixed 512-token chunks, push them into a vector database, and hope the embedding model does the hard work. That hope gets you through a demo. It does not survive contact with real users.

In production, a legal contract falls apart when you sever a liability clause from its exceptions. API documentation turns useless when a code sample gets detached from its function signature. A customer support thread becomes noise when you rip a single complaint out of its conversational history. The problem is rarely the language model sitting at the end of the pipeline. The problem is what you feed it.

We learned this the hard way. Our initial retrieval layer looked standard but behaved inconsistently. So we rebuilt it around a simple idea: treat retrieval as measured infrastructure, not magic. Here is exactly what changed, and how we pushed recall to 95 percent while cutting the 95th-percentile latency from 850 ms to 320 ms.

The Fixed-Chunk Trap

Uniform token counts are easy to code and easy to explain. That convenience masks a basic truth: documents have structure. When you ignore that structure, you destroy signal.

Consider a ten-page master service agreement. A fixed 512-token slice will land mid-obligation, splitting a clause from the very cap table that limits it. The retrieval step then returns half a thought. The generator hallucinates the rest. In API documentation, a chunk that is too large dilutes the embedding with boilerplate headers, burying the specific method a developer needs. In support tickets, a fixed window treats a conversation as a bag of sentences, stripping away the back-and-forth that reveals what actually failed.

We stopped treating chunk size as a hyperparameter we guessed. We started treating it as a mapping exercise between the document type and the information architecture inside it.

Match Your Chunking to the Data

The fix is not one perfect chunk size. The fix is three distinct strategies tuned to three distinct data shapes.

Legal documents now go through recursive splitting. The algorithm first looks for the largest natural boundaries—sections, then subsections, then numbered clauses—and only falls back to smaller splits when necessary. This keeps a termination clause attached to its survival conditions. The retrieval step sees complete logical units, which sharply reduces the model’s temptation to invent missing exceptions.

API and code documentation get structure-aware chunking. Markdown headers, code fences, and parameter tables are parsed as atomic units. We do not split inside a code block. We keep docstrings adjacent to their signatures. The result is that a query for a specific class method retrieves the full context a developer needs: the description, the typed parameters, and the working example.

Support and conversational data use semantic chunking. Instead of counting tokens, we look for shifts in topic or intent. If a customer describes a bug in message three and pastes a stack trace in message seven, we chunk by meaning, not by message index. The retrieval layer then returns the full arc of the problem rather than a orphaned sentence.

Why Vector Search Alone Fails

Even perfect chunks die in a pure vector search. Dense embeddings excel at capturing meaning and synonymy, but they are notoriously fuzzy on exact strings. If an engineer searches for the precise error code ERR_CONNECTION_REFUSED, vector similarity might return a dozen conceptual neighbors and miss the exact match buried at rank fourteen.

Keyword search with BM25 has the opposite problem. It finds exact tokens but misses semantic intent. A user asking “why is my database down” will never match a document that says “troubleshooting connection timeouts.”

We now run both. Vector and keyword results are fed into Reciprocal Rank Fusion, which blends the two ranked lists without requiring calibrated scores. The fused list is then passed through a cross-encoder reranker. The reranker is slower than the initial retrieval, but it is far more precise because it judges query-document relevance directly rather than through compressed embeddings. That hybrid pipeline alone lifted our recall by 15 percent.

Fixing Bad Queries Before They Hit the Index

Users do not write ideal search queries. They paste truncated log lines. They type “it’s broken.” They use jargon your documentation never adopted. If you trust the raw query, you are trusting noise.

We now expand every incoming query into three to five variations before sending them to the retrieval layer. One variation might be a direct paraphrase. Another might be a hypothetical ideal document title. A third strips away conversational filler and isolates technical keywords. Each variant gets embedded and searched. We then deduplicate and merge the candidate pools.

This is not free. Those extra embedding calls cost money and add a few milliseconds. But the effect on recall was dramatic: we moved from 78 percent to 96 percent by expanding queries before retrieval. Because better retrieval shrinks the generation window and grounds the model in correct context, we ended up saving money downstream. A slightly more expensive retrieval step is cheaper than a long, hallucinated generation step.

Stop Guessing. Start Searching.

Once we had the right chunking, hybrid retrieval, and query expansion in place, we still faced a combinatorial mess. Chunk size, chunk overlap, top-k retrieval depth, reranker cutoffs, and fusion weights all interact. A manual grid search would have taken weeks and still left us with a local maximum.

We switched to Bayesian optimization to explore the space. Rather than exhaustively testing every combination, the search algorithm maintains a belief over which configurations are likely to perform well and progressively narrows in on promising regions.

The output is not a single perfect setting. It is a Pareto frontier of choices. On one end, we have a lean configuration optimized for our high-throughput API support endpoint: fast inference, moderate recall, and the lowest possible latency. On the other end, we have an aggressive configuration for legal review: deeper retrieval, heavier reranking, and tighter overlap, trading milliseconds for thoroughness. Because the frontier is explicit, we can pick the right point for the product instead of pretending one size fits all.

What the Numbers Actually Look Like

These changes moved the system from a brittle prototype to a measured production pipeline.

Recall at ten improved from 78 percent to 95 percent. That means when the correct answer exists in our corpus, we find it nineteen times out of twenty.

Latency at the 95th percentile dropped from 850 ms to 320 ms. The hybrid stack sounds heavier on paper, but smarter indexing, smaller rerankers, and the ability to serve aggressive chunks only when needed made the whole system faster.

Hallucination rate—tracked by human annotators on a held-out golden dataset—fell from 12 percent to 3 percent. When the model receives complete, relevant context, it stops inventing facts.

Cost per query dropped from $0.008 to $0.005. Better retrieval means shorter, more focused LLM prompts and fewer recovery attempts. The extra embedding spend on query expansion is dwarfed by the savings in generation.

Build a Golden Dataset and Treat Retrieval Like Code

If you take one thing from this, it should be the discipline of measurement. We built a small golden dataset of real questions and verified answer locations. Before any change hits production, it runs against that dataset. Recall and latency are monitored in real time, not eyeballed in a notebook.

Retrieval is not a research demo. It is infrastructure. It deserves unit tests, regression benchmarks, and automated optimization just like the rest of your stack. Chunk by document structure, not by token superstition. Combine vector and keyword search with a reranker. Expand the queries your users actually write. Then let a search algorithm tune the knobs instead of your intuition.

The pipeline we described is not theoretical. You can read the original write-up here, and if you want to discuss retrieval engineering with a community that cares about this stuff, the GyaanSetu AI group is open.