Check source coverage first.
Ask whether the documents actually contain the answer and whether retrieval returns the relevant section.
RAG troubleshooting
Fix RAG by Finding the Broken Stage, Not by Swapping Models Blindly.
Most RAG problems come from missing source coverage, weak parsing, bad chunk boundaries, wrong metadata, low retrieval recall, no reranking, stale indexes, or prompts that force the model to answer without evidence.
Last updated May 21, 2026. Use this as a debugging loop before changing providers or fine-tuning a model.
Fast diagnosis: if the right chunk is not retrieved, fix parsing, chunking, embeddings, filters, or query rewriting. If the right chunk is retrieved but the answer is wrong, fix reranking, evidence selection, prompt constraints, citations, or the answer model. If old facts appear, rebuild or version the index.
Symptoms
Classify the failure before changing the stack.
Preserve metadata.
Keep document ID, section title, page number, URL, license, and chunk ID with every retrieved passage.
Measure each stage.
Log parsing, embedding, vector search, reranking, model call, and rendering separately.
Create a test set.
Use repeat questions with expected source chunks so you can compare changes without guessing.
| Symptom | Likely cause | What to inspect | First fix |
|---|---|---|---|
| Answer is confident but unsupported | Prompt allows guessing or context is weak | Cited passages, retrieved chunks, abstention rules | Require citations and allow "not enough evidence" |
| Right document never appears | Parsing, chunking, embeddings, or filters | Raw text, chunk boundaries, metadata filters, top-k results | Fix parser and chunking before changing the LLM |
| Right chunk appears too low | Weak ranking after first retrieval | Top-20 candidates and score distribution | Add reranking or hybrid keyword plus vector search |
| Old information appears | Stale index or mixed versions | Index timestamp, document version, source IDs | Version documents and schedule reindexing |
| Good retrieval, bad final answer | Answer model or prompt cannot follow evidence | Prompt, context length, answer schema, model output | Tighten prompt and test another answer model |
Step-by-step
A boring RAG debug loop that works.
Do not tune five things at once. Change one stage, rerun the same questions, and record whether retrieval or answer quality improved.
RAG debug checklist
- Write 20 questions and expected source chunks.
- Inspect parsed text before chunking.
- Check chunk boundaries and metadata.
- Run retrieval without generation and log top-k results.
- Add reranking or keyword search if relevant chunks are buried.
- Force citations and allow abstention in the answer prompt.
- Classify each failure and rerun after one change.
People also ask
RAG troubleshooting questions.
Why does my RAG app hallucinate even with citations?
Citations can be wrong if retrieval returns irrelevant chunks or the answer model cites nearby text without support. Check whether the cited passage truly contains the claim.
Should I increase top-k to fix retrieval?
Sometimes, but increasing top-k can add noise. First inspect top results, improve chunking and metadata, then try reranking or hybrid search.
When should I fine-tune for RAG?
Fine-tune only after source coverage, parsing, chunking, retrieval, reranking, prompting, and evaluation are working. Most early RAG failures are not fine-tuning problems.