How to Build a RAG Pipeline Without Hallucinating Sources

Short version: a RAG pipeline stops inventing sources when citations stop being free text. Retrieve with hybrid search and a reranker, hand the model your chunks as citable blocks with stable IDs, let it say "I don't know", and check every citation against what you actually sent before a user sees it. Each step closes a different gap, and you need all of them.

Most RAG tutorials stop at "embed, retrieve, prompt", which gets you answers that sound sourced. This guide is about answers that are sourced. It builds on the ideas in our explainer on context engineering: what goes into the context window matters more than how cleverly you phrase the prompt.

Sources: Anthropic's Citations, search results and contextual retrieval docs, OpenAI's file search and retrieval guides, Google's Gemini File Search docs, Voyage, Cohere and Ragas documentation, all checked on 29 October 2026. We haven't benchmarked these setups ourselves, so every number below comes from those pages, and the pipeline is the pattern we recommend rather than a lab result.

What You'll Build in This Guide

• A seven-stage RAG pipeline that goes from raw pages to verified, cited answers: ingest, chunk, contextualise, retrieve, rerank, generate, verify.
• Structured citations using Claude's search result blocks, with notes on how OpenAI File Search and Gemini File Search return theirs.
• A citation check in a few lines of Python that rejects any citation pointing outside the passages you supplied.
• Three copy-ready prompts for adding context to chunks, answering only from evidence and auditing claims after the fact.
• An eval loop built on faithfulness, context precision and context recall, so you can tell retrieval failures from generation failures.

Why RAG Pipelines Hallucinate Sources

A hallucinated source rarely comes from one bug. It's usually one of three failures, and each needs a different fix.

1. Retrieval missed the evidence. If the passage that answers the question never reaches the model, the model fills the gap from memory and dresses it up with whatever document is nearby. Anthropic's contextual retrieval post gives a clean example: embeddings can find content about error codes in general but miss an exact match like "Error code TS-999", which keyword search (BM25) catches.

2. The chunk lost its context. Splitting documents strips meaning. Anthropic's example is a chunk that reads "The company's revenue grew by 3% over the previous quarter." On its own it doesn't say which company or which quarter, so it's hard to retrieve and easy to misattribute.

3. Citations were generated, not resolved. When you ask a model to "cite your sources" in plain text, the citation is just more generated text. It can name a document you never sent, or quote a sentence nobody wrote. Too much loosely relevant context makes this worse, the effect we cover in what context rot is and how to fix it.

Pro tip: Before building anything, check whether you need RAG at all. Anthropic notes that if your knowledge base is under about 200,000 tokens (roughly 500 pages), you can put all of it in the prompt and use prompt caching to keep it fast and affordable.

The Pipeline at a Glance

Here's the full flow. Each stage has one job, and each hands the next stage something it can check.

1. Ingest: pull clean source text and give every document a stable ID and URL.
2. Chunk: split documents into passages of a few hundred tokens, keeping the document ID on every chunk.
3. Contextualise: prepend a short, generated description of where each chunk sits in its document.
4. Retrieve: run keyword (BM25) and embedding search in parallel and merge the results.
5. Rerank: score the top candidates with a reranker and keep the best 10 to 20.
6. Generate: pass the survivors to the model as citable blocks and allow "not in the sources".
7. Verify: check every citation against the passages you sent, then score the answer in your eval set.

Step 1: Ingest Clean Sources With Stable IDs

Garbage in the index means garbage in the answer. Navigation menus, cookie banners and footers get chunked and embedded like real content, and they compete with real content at retrieval time. Strip them before indexing.

For web sources, Firecrawl handles the cleaning for you. Its scrape endpoint returns Markdown with onlyMainContent on by default, and you can cut more with includeTags and excludeTags. To index a whole docs site, its crawl endpoint charges 1 credit per page and defaults to a 10,000-page limit, so set limit to the size you actually want. The full setup with Claude, including the cleaning step, is in this morning's guide to using Firecrawl with Claude.

Whatever the source, store three things for every document:

• A stable ID that never changes when you re-index, such as a URL or an internal key like kb://pricing-faq.
• A title readers will recognise when it shows up next to a citation.
• Metadata you may filter on later: product, language, version, last-updated date.

Stable IDs are what make verification possible. If an ID is regenerated on every run, you can't tell a real citation from a stale one.

Step 2: Chunk Documents and Add Context to Every Chunk

Start with sensible defaults rather than inventing your own. OpenAI's retrieval guide says vector stores split files into 800-token chunks with 400 tokens of overlap by default. You can change it per file, with a chunk size between 100 and 4,096 tokens and overlap no more than half the chunk size. Gemini File Search exposes the same idea through chunking_config, with a maximum of tokens per chunk and overlapping tokens. Anthropic describes typical chunks as "no more than a few hundred tokens".

Then fix the lost-context problem. Contextual retrieval asks a model to write a short note for each chunk explaining where it sits in the full document, and prepends that note before embedding and before building the keyword index. Anthropic reports the note is usually 50 to 100 tokens. The revenue chunk from earlier becomes "This chunk is from an SEC filing on ACME Corp's performance in Q2 2023..." followed by the original sentence.

Running a model over every chunk sounds expensive, but the whole document is the same for every chunk in it, so it caches well. With prompt caching, Anthropic estimated a one-time cost of $1.02 per million document tokens for its setup (800-token chunks, 8,000-token documents). If you haven't set caching up yet, our guide to Claude prompt caching covers the mechanics. A smaller, cheaper model is a reasonable choice for this job: Anthropic used Claude 3 Haiku in its experiments.

This is our adaptation of the contextualiser prompt. Run it once per chunk, with the full document cached:

Keep the original chunk text separate from the generated note in your store. Retrieval should see both, but citations should point at the original text, because that's what a reader will check.

Step 3: Retrieve With Hybrid Search, Then Rerank

Embeddings find meaning; BM25 finds exact strings like product codes, error messages and names. Run both and merge the result lists with rank fusion. If you use OpenAI File Search, the tool already searches with semantic and keyword matching, and its ranking_options let you set a score_threshold between 0 and 1 and tune the embedding and text weights of its reciprocal rank fusion.

Two details matter when you embed your own chunks. First, pick an embedding model built for retrieval. Anthropic doesn't offer its own embedding model and points developers to Voyage AI, whose current lineup includes voyage-4-large, voyage-4 and voyage-4-lite with a 32,000-token context. Second, tell the model which side of the search each text is on: Voyage recommends setting input_type to query or document for retrieval.

Then rerank. A reranker reads the question and each candidate together and scores relevance, which is slower than vector similarity but more precise. Anthropic's summary of its own tests is blunt: "Reranking is better than no reranking." Anthropic's experiments retrieved the top 150 candidates, reranked them with Cohere's reranker and passed the top 20 to the model. Cohere's current top model is rerank-v5.0-pro, with a faster rerank-v5.0-fast; Voyage offers rerank-3 and rerank-3-lite.

Here's what each layer was worth in Anthropic's tests, measured as the share of questions where the relevant chunk was missing from the top 20:

Honest answer: those are Anthropic's numbers on its own datasets from 2024, not a guarantee for yours. The direction is what to take away: every layer helped, and the gains stacked.

Step 4: Pass Chunks as Citable Blocks, Not Pasted Text

This is the step most tutorials skip, and it's the one that stops invented sources. Don't paste retrieved chunks into the prompt as one long string and ask for "[1]"-style citations. Give the model each chunk as a structured block with its source attached, and let the API return citations as data.

On Claude, the tool for this is the search result content block. Each block carries a source (a URL or any stable ID such as kb://article-1234), a title and an array of text blocks. You can return them from a custom search tool or put them straight into the user message:

{
  "type": "search_result",
  "source": "kb://pricing-faq",
  "title": "Pricing FAQ",
  "content": [
    {"type": "text", "text": "Annual plans renew on the purchase date."},
    {"type": "text", "text": "Refunds are available within 14 days of renewal."}
  ],
  "citations": {"enabled": true}
}

When Claude uses that content, its answer comes back as text blocks with a citations list. Each citation has type search_result_location and carries the source, title, the cited_text and the exact block range it came from. Each text block is the smallest unit Claude can cite, so split long passages into several blocks if you want sentence-level citations.

Why this beats prompting: Anthropic's Citations docs say API citations are guaranteed to contain valid pointers to the provided documents, and that in Anthropic's evaluations they were significantly more likely to cite the most relevant quotes than prompt-based citations. The cited_text field doesn't count toward output tokens either.

If you expose retrieval as a tool, Claude decides when to search. That's the same design choice we weigh in Claude skills vs MCP tools vs subagents. For a support bot where every answer needs evidence, force the call with tool_choice, as Anthropic's own example does.

The other platforms have their own versions:

• OpenAI File Search returns file_citation annotations with the file ID and filename. It doesn't return the retrieved chunks by default; add include: ["file_search_call.results"] so you can verify them.
• Gemini File Search returns citations in the annotations of the model's text output, showing which parts of your uploaded documents were used.

Step 5: Write an Answer Prompt That Allows "I Don't Know"

Structured citations stop fake pointers. They don't stop the model from answering beyond the evidence. That's the prompt's job, and Anthropic's guide to reducing hallucinations lists the levers: give the model explicit permission to say it doesn't know, restrict it to the provided documents, and for long material, have it extract word-for-word quotes before it answers.

The answer policy we recommend for a grounded assistant:

Rule 4 matters more than it looks. Retrieved pages are untrusted input, and a scraped page can contain text written to steer a model. Your application decides what counts as an instruction, not the documents.

Pro tip: Test the refusal path on purpose. Put questions in your eval set whose answers are not in the knowledge base, and count a confident answer to any of them as a failure, even if it happens to be true.

Step 6: Verify Every Citation Before the User Sees It

Trust, then check. Two checks catch different problems, so run both.

Check 1: does each citation point at something you sent? This one is deterministic and costs no tokens. Keep the exact search results you passed to the model, then confirm every returned citation names one of those sources and that its cited_text matches the blocks it points at. A minimal version in Python, assuming the SDK response from Step 4:

sent = {r["source"]: r for r in search_results}   # exactly what you passed in

def check_citations(response):
    problems = []
    for block in response.content:
        if block.type != "text":
            continue
        for c in (block.citations or []):
            doc = sent.get(c.source)
            if doc is None:
                problems.append(("unknown source", c.source))
                continue
            cited = doc["content"][c.start_block_index:c.end_block_index]
            if not all(t["text"] in c.cited_text for t in cited):
                problems.append(("text mismatch", c.source))
    return problems

With API citations this should almost never fire, which is the point. When it does, you've found a bug in your plumbing (re-indexed IDs, a stale cache, a wrong source string), not a model problem.

Check 2: does each cited passage actually support its claim? A real citation next to an overstated sentence is still a hallucination. Anthropic suggests having the model find a supporting quote for each claim after it writes, and retract any claim it can't support. Run this as a separate call, ideally with a fresh context, the same reviewer pattern covered in our guide to multi-agent workflows with Claude Code:

Ship the corrected answer, log the labels, and send anything with a NOT SUPPORTED claim to your review queue.

Step 7: Measure Faithfulness and Retrieval With Evals

You can't tune what you don't measure, and "the answers look good" is not a measurement. Build a test set of a few dozen real questions with the passages that answer them, plus a handful of questions the knowledge base can't answer. Then score three things separately, using definitions from Ragas:

• Faithfulness (generation): the number of claims in the answer supported by the retrieved context, divided by the total number of claims. It runs from 0 to 1; anything below 1 means the model said something the sources don't back.
• Context precision (ranking): whether the relevant chunks are ranked near the top of what you retrieved.
• Context recall (retrieval): how many of the relevant pieces of information you retrieved at all.

Read them together. Low recall with high faithfulness means the model is behaving and your retrieval is starving it, so revisit chunking, context and hybrid search. High recall with low faithfulness means the evidence is there and the model is drifting, so tighten the answer prompt and the claim audit. Re-run the set whenever you change the embedding model, chunk size, reranker or prompt.

Tool Stack: Managed File Search vs Build Your Own

You don't have to assemble every piece yourself. Managed file search handles chunking, embeddings and storage; building your own gives you control over hybrid search, reranking and chunk context. Here's how the main options compare on the details that affect citations:

Our take: start managed if you need a working prototype this week, and move retrieval in-house once your eval set shows retrieval is the bottleneck. You can keep the Claude citation format either way, since search result blocks accept anything your own search returns.

For the bulk jobs in this pipeline, writing chunk context and running claim audits, a lower-cost model is worth trying before you default to a frontier one. GLM models through Z.ai are one option to evaluate against your test set, and our guide on how to choose thinking level and effort covers when a lower reasoning setting is enough.

Common Pitfalls That Bring Hallucinations Back

• Combining citations with structured outputs. On Claude, enabling citations on any document or search result while also setting output_config.format returns a 400 error. If you need JSON, generate the cited answer first, then convert it in a second call.
• Mixing citation settings. Citations must be enabled on all documents in a request or none, and the same rule applies to search results. Mixing them returns an error.
• Raising errors on empty searches. Anthropic recommends returning a plain text block such as "No results found." so the model can tell the user, rather than failing or improvising.
• Citing scanned PDFs. Claude cites text, not images, so a scanned PDF without extractable text can't be cited. Run OCR before indexing.
• Verifying against nothing. OpenAI File Search doesn't return the retrieved chunks unless you ask with include. Without them, you can't run Check 1.
• Expecting web grounding and file search together. Gemini File Search can't be combined with Grounding with Google Search or URL Context in the same request. Route web questions and knowledge-base questions separately.
• Stuffing in more chunks to be safe. Anthropic found 20 chunks beat 10 and 5 in its tests, and also warned that more information can distract the model. Treat chunk count as a setting to evaluate, not a free win.

Verdict: The Pattern We Recommend

My take: hallucinated sources are a design problem, not a model problem. If citations are free text, any model will eventually invent one. If citations are structured pointers into passages you control, and you check them, fake sources become a bug you can catch in code.

The minimum version worth shipping has four parts: hybrid retrieval with a reranker, chunks passed as search result blocks with citations enabled, an answer prompt that allows "I couldn't find that", and the deterministic citation check. Add contextual chunks, the claim audit and a Ragas eval set as soon as you have real traffic. Each piece is small; together they turn "sounds sourced" into "is sourced".

Keep Reading

• 12 Context Engineering Patterns for Claude Code Agents
• Gemini Implicit Caching vs Claude Explicit Caching
• How to Build Multi-Agent Workflows With Gemini Managed Agents