Most teams still assemble their first retrieval pipeline the same way. They pick a fixed token limit, maybe 512, split documents into uniform blocks, and feed those blocks into a vector database. On a small dataset with simple questions, this looks magical. In production, it falls apart.
Legal contracts shred into meaningless fragments when a clause gets sliced mid-sentence. API documentation turns into noisy soup if a single chunk swallows three unrelated functions. Customer support tickets lose all narrative thread without overlap between segments. The result is predictable: bloated latency, weak recall, and answers that force the generator to hallucinate.
We tore our retrieval layer down and rebuilt it. The outcome was a jump in recall from 78 percent to 95 percent, a 62 percent cut in latency, and a pipeline that finally behaves like real infrastructure instead of a weekend hack. Here is what actually worked.
Smart Chunking: Structure Over Tokens
The first mistake is assuming every document speaks the same language. A 512-token chunk makes sense for narrative prose and almost nowhere else. We moved to a strategy that respects the anatomy of the source.
For legal documents, we use recursive chunking. The algorithm first tries to split on high-level boundaries like sections and articles. If a section is still too long, it looks for subsections, then paragraphs, then sentences. This preserves the logical nesting of clauses. A non-compete agreement stays intact. Definitions do not bleed into indemnity terms.
API documentation demands structure-aware chunking. A function signature, its parameter table, and its example request belong together. Splitting after a fixed token count often leaves the parameters in one chunk and the examples in another. We chunk by document object instead. One chunk holds a complete endpoint or a single function. The retriever can then return a self-contained reference that actually answers the question.
Support tickets naturally suit semantic chunking. Instead of cutting at a token boundary, we detect where the topic shifts. A ticket that starts with a login complaint and pivots to a billing question splits into two coherent pieces. Each piece carries the metadata it needs, and the model no longer has to guess which problem the user actually cares about.
Internal wikis are messier. They mix prose, tables, diagrams, and embedded threads. For these, we use agentic chunking. A small language model reads ahead and decides where a thematically complete unit ends. It costs a bit more at ingestion time, but it eliminates the human busywork of hand-tuning rules for every new page format.
Hybrid Retrieval: Cover Your Bases
Vector search is excellent at capturing fuzzy meaning. Ask about slow uploads and it will happily return paragraphs about latency and bandwidth. But it is notorious for mangling exact matches. If a developer searches for the error code ERR_CONNECTION_REFUSED, dense embeddings will often treat it as generic noise.
BM25, the classic keyword algorithm, does the opposite. It nails precise strings and rare terms, yet it misses semantic nuance. A query about signing the agreement might never surface content tagged with executing the contract.