Production RAG at Scale: Lessons from 10,000+ Listings Daily
I built a RAG pipeline for a job board. It worked in staging but struggled under real load. Processing thousands of listings daily requires more than a good vector store. You must understand where the system breaks.
Here are my lessons on chunking, embeddings, costs, and observability.
- Don’t guess your chunking strategy
Most tutorials treat chunking as a simple setting. In production, your strategy dictates accuracy and cost.
I tested three methods for job listings:
- Fixed-size chunks: These failed. They split sections like "requirements" and "benefits" at random points. This creates noisy retrieval.
- Semantic chunking: This was better but inconsistent. Some chunks were too long and others were too short.
- Recursive character splitting with overlap: This worked best. I split on newlines and sentences. I used a 400 token size with a 50 token overlap. This ensures sentences spanning two chunks stay connected.
Pro tip: Normalize your data before chunking. Different sources like Greenhouse or Lever return different formats. Clean the text first so your chunker sees a consistent structure.
- Embeddings: Cost vs. Accuracy
I tested Llama 3.1 via Ollama against OpenAI text-embedding-3-small. The local model was free but struggled with domain-specific terms like "equity compensation." It produced noisy results. OpenAI cost more but provided accurate matches. I chose OpenAI because bad retrieval costs more in LLM calls later.
To save time, I batch my requests. I send up to 100 chunks in one call. This reduces latency and keeps the pipeline fast.
- The Vector Store Trade-off
I used Pinecone for prototyping because it is fast to set up. However, at scale, the costs grew too high.
I switched to pgvector inside PostgreSQL.
- It was more work to set up.
- It saved massive amounts of money.
- It provided transactional consistency. Since the embeddings live in the same database as the job data, you have one source of truth. You do not need to sync two different systems.
- Controlling LLM Costs
Scoring every listing with GPT-4o is expensive. I used three tactics to lower costs:
- OpenAI Batch API: I process scoring jobs overnight. This provides a large discount.
- Caching: I cache results for repeat candidate profiles.
- Model tiering: I use GPT-4o-mini for common roles like "Sales Representative." I only use GPT-4o for niche roles where precision is vital.
- Build Observability First
My pipeline once failed silently. Malformed data caused empty chunks, which the system skipped without error.
I fixed this by adding structured logging with a correlation ID. This allowed me to trace one listing from ingestion to scoring. I could finally see which data sources were causing failures.
The biggest lesson: Most problems come from messy data, not the AI. Fix your data plumbing first.
Optional learning community: https://t.me/GyaanSetuAi