How RAG Works
RAG splits the generation process into two stages. First, a retrieval system searches your document store (vector database, search index, or API) for chunks relevant to the user query. Second, the LLM receives those chunks as context and generates a response grounded in the retrieved information.
The retrieval step typically uses embeddings — dense vector representations of text — stored in a vector database like Pinecone, Weaviate, or pgvector. When a query arrives, it gets embedded using the same model, and the nearest neighbors in vector space are returned as context.
When to Use RAG
RAG works best when your data changes frequently and retraining or fine-tuning the model is impractical. Common use cases: customer support bots that need current product documentation, internal knowledge bases with company-specific policies, and research assistants that need to cite specific sources.
RAG is overkill for static, well-known information that the model already handles correctly. If GPT-4 or Claude already gives accurate answers to your queries without context, adding a retrieval layer adds latency and complexity for no benefit.
Common RAG Pitfalls
The biggest failure mode is poor chunking. If your documents are split at arbitrary character limits, the retrieved chunks may contain half-sentences that confuse the model. Semantic chunking — splitting at paragraph or section boundaries — produces significantly better results.
The second failure mode is embedding model mismatch. Using a general-purpose embedding model on domain-specific jargon (medical, legal, financial) often returns irrelevant chunks. Domain-tuned embeddings or hybrid search (vector + keyword BM25) fix this.