Most RAG tutorials end at the notebook. They load a few polished PDFs, split the text every thousand characters, stuff the fragments into a vector database, and call it an architecture. On a Friday afternoon, that demo runs perfectly. In production, the same pipeline quietly turns into a liability.

The real bottleneck in a retrieval system is rarely the model or the prompt. It is ingestion. A RAG pipeline can only retrieve what it has been fed, and if the feed is noisy, stale, or incomplete, the model will deliver confident nonsense. When users complain that the bot hallucinated, the fault often lies miles upstream in a data pipeline that nobody monitors closely.

The Whiteboard Trap

Architecture diagrams make ingestion look like a single arrow labeled “Documents → Vector DB.” Reality is messier. Source systems change without notice. HTML layouts get redesigns. URLs redirect to generic landing pages. JavaScript frameworks swap out the content after the initial HTTP response. Treating ingestion as a one-time setup task is the first mistake. It is an ongoing data engineering problem that deserves the same rigor as any ETL pipeline.

Why RAG Failures Are Usually Feed Failures

Picture this: a user asks your internal assistant about the current refund policy. The model pulls the top chunk from the vector store and states a 30-day window. The actual policy changed to 60 days last quarter. The LLM did not invent the wrong answer. It trusted bad input. The retrieval layer served an old page, and because the embedding looked semantically close enough, the model treated it as ground truth.

This pattern repeats constantly. Teams burn hours tweaking temperature and top-k when their corpus is full of navigation footers, duplicate press releases, and chunks that split tables in half. Before you optimize generation, audit what your system is allowed to know.

Seven Traps That Destroy Ingestion

1. The First Run Is a Lie

A green checkmark on your initial crawl means almost nothing. Production data is alive. Documentation pages get refactored, blog permalinks break, and sitemaps quietly drop sections. If you only validate that the pipeline completed without throwing an error, you are flying blind. You need to validate the output. Check that the expected documents are present, that their structure still parses, and that the total volume of text hasn't collapsed because a source decided to paginate results differently.

2. Crawling Is Not Ingestion

Fetching HTML is the easy part. A raw crawl captures everything: cookie banners, "Related Articles" sidebars, ad blocks, and footer copyright notices. If you chunk that raw HTML naively, every single piece of text carries along fragments of the navigation menu. When a user asks about API rate limits, the retriever might surface a chunk that is 40 percent sidebar links. Clean extraction matters. You need to identify the main content area, strip boilerplate, and remove elements that repeat across every page. Otherwise you are not building a knowledge base. You are building a search engine for website chrome.

3. Chunking Breaks Meaning

Fixed-size chunking is the default in nearly every quickstart guide, and it is dangerous. Split a document purely by character count and you will slice tables down the middle, separate steps 4 and 5 in a numbered procedure, and orphan bullet points from their headings. A chunk containing only the second half of a pricing table is semantically useless. Structure-aware chunking respects the original format. Parse the heading hierarchy. Keep tables intact where possible. Split at paragraph boundaries under the same H2 or H3. Preserve lists inside a single chunk if they are short enough. The goal is not evenly sized blocks. The goal is coherent units of meaning.

4. The Freshness Problem

A static snapshot of an internal wiki is simple mode. Continuously ingesting from the live web is hard. You need to know when a page was last gathered, whether it has changed since then, and how long the information remains valid. Stale data does not always mean a visibly old date. Sometimes a page updates its text but keeps the same URL, so your system never notices without content hashing. Build clear refresh rules based on source volatility. A financial data feed might need hourly checks. A company about page might need quarterly checks. Record timestamps and set time-to-live boundaries, especially if your domain involves regulated or safety-critical guidance where old facts can cause real harm.

5. Duplicate Pollution

Websites are full of repetition. The same product description appears on the category page, the product page, and a promotional landing page. The same press release lives under /news/, /press/, and /blog/. Vector search does not deduplicate automatically. If ten nearly identical chunks sit in your database, they can crowd out diverse, relevant results in your top-k retrieval. You need canonical tracking or content deduplication before embedding. If two chunks say the same thing, keep the authoritative source and drop the copies. Your retriever has limited slots. Do not let them go to waste.

6. Missing Metadata

A vector database without metadata is just a dense text search engine with no memory of context. Smart retrieval depends on filtering and ranking signals that raw embeddings cannot provide. Store the source URL, the capture date, the document category, and the version number. If you ingest API documentation, versioning is essential. Without it, a query might blend v1 and v2 specs into the same answer. If you ingest HR policies, tagging by region or department lets you filter results before they ever reach the model. Metadata turns a text dump into a curated knowledge system.

7. JavaScript Gaps

Modern sites do not ship their content in the first HTML payload. They send a skeleton and hydrate it with JavaScript calls. A basic HTTP request might see nothing but a loading spinner and a layout shell. If your pipeline cannot execute JavaScript, you will ingest blank pages or partial fragments and never realize anything is wrong. Using a headless browser solves the rendering problem but introduces new ones: heavier memory use, slower throughput, and bot detection walls. Choose your trade-offs deliberately, but do not pretend a simple curl equivalent is enough for every source.

A Practical Ingestion Checklist

If you are building or reviewing a RAG feed, start here:

  • Validate source coverage and pagination. A sitemap might list only the first ten articles in a category. Crawl deep and verify that paginated or dynamically loaded content is actually captured.
  • Remove boilerplate before chunking. Strip navigation, ads, footers, and repeated legal disclaimers. If a phrase appears on every page, it is noise.
  • Use structure-aware chunking. Respect headings, bullet lists, and tables. Split on semantic boundaries, not character counts.
  • Attach rich metadata. Include URL, capture date, content category, and version. Make these fields filterable in your retrieval queries.
  • Set refresh frequencies based on data volatility. High-change sources need frequent re-crawls. Static archives do not.
  • Monitor the corpus, not just the job status. A pipeline can exit with code zero while producing garbage. Audit samples of stored chunks regularly for drift and quality.
  • Define rules for versioning and deletions. When a source page is removed, delete its chunks. When it updates, overwrite or version them. Orphaned data is a silent killer.

The Hard Truth About Embeddings

No embedding model, no matter how advanced, can repair a missing document. It cannot guess that a page was updated last week if your feed still holds last year's copy. It cannot infer the context of a table row that got separated from its header by a bad chunk boundary. Embeddings compress meaning, but they do not create meaning where the ingestion layer failed to preserve it.

Retrieval quality starts at the ingestion layer. That layer decides whether your RAG system is a useful tool or just a confident liar with a vector database behind it.

The Real Takeaway

Stop measuring ingestion health with pipeline dashboards alone. Green jobs and clean logs do not guarantee a clean corpus. Open the database and read the actual chunks your users will retrieve. If the text is full of copyright notices, split tables, and outdated policy pages, your problem is not the LLM. Fix the feed first. Everything else is tuning on top of garbage.