RAG Explained: How to Make AI Remember Your Documents

Building a Complete RAG System

The Full Pipeline

// Complete RAG pipeline in ~50 lines
import { ChatAnthropic } from '@langchain/anthropic';
import { OpenAIEmbeddings } from '@langchain/openai';
import { MemoryVectorStore } from 'langchain/vectorstores/memory';
import { RecursiveCharacterTextSplitter } from 'langchain/text_splitter';
import { createRetrievalChain } from 'langchain/chains/retrieval';
import { createStuffDocumentsChain } from 'langchain/chains/combine_documents';
import { ChatPromptTemplate } from '@langchain/core/prompts';

// 1. Load and split documents
const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 1000,
  chunkOverlap: 200,
});
const splits = await splitter.splitDocuments(docs);

// 2. Create vector store
const vectorStore = await MemoryVectorStore.fromDocuments(
  splits,
  new OpenAIEmbeddings()
);

// 3. Create retriever
const retriever = vectorStore.asRetriever({ k: 4 });

// 4. Create the chain
const llm = new ChatAnthropic({ model: 'claude-sonnet-4-6-20250320' });
const prompt = ChatPromptTemplate.fromTemplate(`
  Answer based on the context. If unsure, say so.
  Context: {context}
  Question: {input}
`);

const chain = await createRetrievalChain({
  combineDocsChain: await createStuffDocumentsChain({ llm, prompt }),
  retriever,
});

// 5. Query!
const result = await chain.invoke({
  input: "What is the refund policy?"
});
console.log(result.answer);

---

Common RAG Pitfalls and Solutions

Pitfall 1: Poor Chunk Quality

Problem: Chunks split mid-sentence or mid-paragraph, losing context.

Solution: Use semantic chunking that respects document structure. Split on paragraphs and sections, not arbitrary character counts.

Pitfall 2: Irrelevant Retrieval

Problem: The system returns chunks that are semantically similar but not actually relevant.

Solution: Implement hybrid search — combine vector similarity with keyword matching. Add metadata filtering (date, source, category) to narrow results.

Pitfall 3: The AI Ignores the Context

Problem: The LLM answers from its training data instead of the provided context.

Solution: Strengthen the system prompt. Use explicit instructions like “ONLY use the provided context” and “If the answer isn’t in the context, say ‘I don’t have that information.'”

Pitfall 4: Context Window Overflow

Problem: Too many retrieved chunks exceed the model’s context window.

Solution: Implement reranking to select only the most relevant chunks. Use a two-stage retrieval: broad retrieval followed by a reranker model.

---

People Also Ask

Do I need a vector database for RAG?

For production systems, yes. For prototyping and small datasets (under 10,000 documents), in-memory vector stores work fine. Popular choices: Pinecone, Qdrant, Weaviate, Chroma, and Supabase pgvector.

How much does RAG cost to run?

The main costs are: embedding generation ($0.13 per million tokens for OpenAI), vector database hosting ($25-$100/month for managed services), and LLM inference for answers. For a small to medium knowledge base, expect $50-$200/month total.

Can RAG work with private/sensitive documents?

Yes, and this is one of its biggest advantages. You can run RAG entirely on-premise using open-source components (Ollama for the LLM, Chroma for vectors). Your documents never leave your servers.

---

Next Steps

1. Start small — build a RAG system over 10-20 documents first
2. Use LangChain or LlamaIndex — don’t build from scratch
3. Measure retrieval quality — track if the right chunks are being found
4. Iterate on chunking strategy — this is often the biggest lever
5. Add evaluation — test with known questions and expected answers

RAG is not a set-and-forget system. The best implementations are continuously refined based on user feedback and retrieval metrics.

---

Want to skip months of trial and error? We’ve distilled thousands of hours of prompt engineering into ready-to-use prompt packs that deliver results on day one. Our packs at wowhow.cloud include battle-tested prompts for marketing, coding, business, writing, and more — each one refined until it consistently produces professional-grade output.

Blog reader exclusive: Use code BLOGREADER20 for 20% off your entire cart. No minimum, no catch.

Browse Prompt Packs →