RAG Pipeline: Chunking, Retrieval, Pitfalls

What RAG is

RAG (retrieval-augmented generation) combines search with a language model: relevant passages from your own documents are found for the question first, and the model answers from them. It addresses two weaknesses of an LLM on its own — it does not know your internal data and cannot say where an answer came from. This page is a guide to building and debugging such a pipeline.

What this covers

Designing and debugging a retrieval-augmented generation (RAG) pipeline: splitting source documents into chunks, retrieving relevant ones (dense, sparse, or hybrid), reranking results before they reach the model, and evaluating whether the generated answer is actually grounded in what was retrieved. For embeddings/vector-search fundamentals and local model serving, see AI Development and Ollama .

When to use it

• Designing or debugging a RAG pipeline end to end
• Choosing a chunking strategy and chunk size/overlap for a set of source documents
• Deciding between dense, sparse (keyword), or hybrid retrieval for a search feature
• Adding a reranking step because top-k retrieval keeps missing the right passage
• Evaluating whether a RAG system’s answers are actually grounded in retrieved context

RAG Architecture

A RAG pipeline has two phases:

Ingestion (offline): load documents → split into chunks → embed each chunk → store vectors + text in a vector store.

Query (online): embed the user’s question → retrieve the top-k most relevant chunks → optionally rerank them → pass the retrieved text plus the question to an LLM → generate the answer.

Documents → Chunking → Embedding → Vector Store
                                        │
User Query → Embedding → Retrieval ────┘
                              │
                          Reranking (optional)
                              │
                    LLM Generation → Answer

Chunking Strategies

How you split source text directly affects retrieval quality — chunks too large dilute relevance signal, chunks too small lose context.

Fixed-size with overlap:

def chunk_text(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]:
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start += chunk_size - overlap
    return chunks

Recursive splitting (tries paragraph → sentence → word boundaries, in order, so chunks break on natural text boundaries where possible):

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_text(document_text)

Guidance:

• Start around 300–800 characters (or ~100–300 tokens) with 10–20% overlap; tune against your own retrieval evaluation, not a fixed rule
• Prefer splitting on natural boundaries (paragraphs, sections, headings) over arbitrary character counts when the source format allows it
• Keep metadata (source document, section title, page number) attached to each chunk — it’s needed for citations and for filtering at query time

Retrieval Methods

Dense retrieval (embedding similarity)

import numpy as np

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def dense_retrieve(query_embedding, chunk_embeddings, chunks, top_k=5):
    scores = [cosine_similarity(query_embedding, emb) for emb in chunk_embeddings]
    ranked = sorted(zip(scores, chunks), key=lambda x: x[0], reverse=True)
    return ranked[:top_k]

Good at matching meaning and paraphrases; weak on exact terms it’s never seen (product codes, acronyms, rare proper nouns).

Sparse retrieval (keyword / BM25)

from rank_bm25 import BM25Okapi

tokenized_chunks = [chunk.lower().split() for chunk in chunks]
bm25 = BM25Okapi(tokenized_chunks)

query_tokens = query.lower().split()
scores = bm25.get_scores(query_tokens)
top_indices = np.argsort(scores)[::-1][:top_k]

Good at exact-term matches; misses semantically related text that doesn’t share vocabulary.

Hybrid retrieval (reciprocal rank fusion)

Combine both rankings instead of picking one:

def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[str]:
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

k=60 is the commonly used default from the original RRF paper — it dampens the influence of any single high rank, so a document that’s #1 in one ranking and unranked in the other doesn’t automatically dominate.

Reranking

A cross-encoder scores each (query, chunk) pair jointly, which is more accurate than embedding similarity alone but too slow to run over an entire corpus — so it’s applied only to the retriever’s top-k candidates:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [(query, chunk) for chunk in retrieved_chunks]
scores = reranker.predict(pairs)
reranked = [chunk for _, chunk in sorted(zip(scores, retrieved_chunks), reverse=True)]

Typical pattern: retrieve top 20–50 candidates cheaply (dense/sparse/hybrid), then rerank down to the top 3–5 that actually go into the prompt.

End-to-End Example

Tying chunking, dense retrieval, and generation together with a local Ollama model (see Ollama for setup):

import ollama
import numpy as np

def embed(text: str) -> np.ndarray:
    result = ollama.embed(model="mxbai-embed-large", input=text)
    return np.array(result["embeddings"][0])

def answer(question: str, chunks: list[str], top_k: int = 3) -> str:
    chunk_embeddings = [embed(c) for c in chunks]
    query_embedding = embed(question)

    ranked = dense_retrieve(query_embedding, chunk_embeddings, chunks, top_k=top_k)
    context = "\n\n".join(chunk for _, chunk in ranked)

    prompt = f"Answer the question using only the context below.\n\nContext:\n{context}\n\nQuestion: {question}"
    response = ollama.chat(model="llama3.2:1b", messages=[{"role": "user", "content": prompt}])
    return response["message"]["content"]

Evaluation

RAG quality has two independent failure modes — bad retrieval (the right chunk was never found) and bad generation (the model ignored or misread good context) — so measure them separately:

• Context precision / recall: of the chunks retrieved, how many were actually relevant, and of the relevant chunks that exist, how many were retrieved
• Faithfulness / groundedness: does the generated answer only state things supported by the retrieved context, or does it add unsupported claims
• Answer relevance: does the answer actually address the question, independent of whether it’s grounded

RAGAS is a widely used open-source library for computing these metrics against a labeled evaluation set.

Common mistakes

Best Practices

1. Evaluate chunking and retrieval separately from generation — a bad answer might mean the right chunk was never retrieved, not that the model reasoned poorly
2. Log retrieved chunks alongside every answer in production; without them you can’t tell whether a bad answer was a retrieval or generation failure
3. Re-tune chunk size and top-k against your own evaluation set — defaults that work for one corpus don’t transfer automatically
4. Cache embeddings for unchanged source documents; re-embedding the whole corpus on every ingestion run is wasted cost
5. Attach source metadata to every chunk so answers can cite where they came from