In this article
The essential idea
If the correct passage never reaches the model, changing the prompt rarely solves the problem. Evaluate retrieval separately: exact terms, semantic matches, document structure and ranking each address different failure modes.
Separate the questions that look alike
“How do I reset this device?” and “What does error AX-204 mean?” both look like support questions. The first can benefit from semantic matching when the manual uses different wording. The second depends on preserving an exact identifier. Build a small retrieval test set with both styles, plus acronyms, multilingual questions and requests involving several documents.
For each question, have a subject-matter expert identify the passages that should be found. Inspect retrieval results before asking a model to write an answer. Otherwise, fluent generation can hide that the wrong product manual was selected. Keep a simple lexical baseline; complexity earns its place only when it fixes a failure you can name.
Combine lexical precision and semantic coverage
Hybrid search combines full-text and vector retrieval. Azure AI Search documents an implementation that merges parallel result lists using reciprocal rank fusion. The general engineering opportunity is to retain exact matches while finding passages expressed in different words. It is an approach to evaluate, not proof that every hybrid configuration beats every vector search.
In our example, keep the product and permission filters consistent across both retrieval paths. Inspect which path contributed each useful result, and whether repeated copies of the same document crowd out other evidence. Compare against your baseline on identical questions and source versions. A better average score can still hide a regression on the exact error codes that support staff use most.
Two ways to find the same evidence
Illustrative support questions. Compare both paths on your own documents.
Exact words
“AX-204” → preserve the error code, product name and identifiers.
Meaning
“The device will not restart” → find a differently worded procedure.
Combined evidence
Merge candidates, remove duplicates and evaluate the final ranking.
Make each chunk understandable in isolation
A chunk is the unit of text sent through retrieval. We recommend starting from document structure: a section with its heading, a complete procedure, or a table with its column labels. Splitting every document at an identical character count can detach an exception from the rule it qualifies. Excessive overlap creates duplicates and wastes context.
There is no universal chunk size we would prescribe before seeing the material. Test a few approaches on representative documents, including scanned PDFs and long tables. Preserve a link to the original location and enough neighbouring context to resolve references such as “the following conditions”. If extraction has lost a table header, a more expensive model cannot reliably reconstruct it.
Add reranking only with a measurable benefit
Reranking is a second ordering step applied to retrieved candidates before choosing the final context. Our suggested experiment is to hold the retriever constant and compare the evidence selected with and without this step. Measure relevant passages near the top, missing evidence, latency and cost. A reranker cannot rescue a source excluded by an incorrect filter.
Keep the pipeline modular so that query preparation, retrieval and context assembly can be inspected independently. Spring AI’s RAG documentation illustrates this modular approach with its advisor APIs. Before expanding the pipeline, write down the observed defect and the expected improvement. Ship the simpler configuration if the added stage does not improve the questions that matter.
Debug an exact reference and a natural-language question
Use two illustrative queries: “Error E-417 on controller X2” and “the controller stops when the room gets hot”. The first depends on preserving an exact identifier; the second may need meaning-based retrieval because the manual calls the issue “thermal shutdown”. A retrieval test set should contain both, not just polished natural-language questions.
Inspect the candidate passages before looking at the final answer. If the right manual never entered the candidate set, a reranker cannot recover it. Check identifier tokenisation, filters, language and document coverage first. If the right passage is present but buried beneath less relevant material, ranking is a plausible target. This sequence avoids changing three stages at once and losing the explanation for any improvement.
Use a failure matrix instead of one aggregate score
Group questions by failure type: exact code, paraphrase, missing source, ambiguous product and conflicting revision. Compare configurations on the same groups and retain the retrieved passages. A single average can hide a regression on the reference codes that support staff use most often.
Measure response time alongside relevance and the amount of context sent to generation. More retrieved passages can add cost and contradictory material without improving the answer. Choose a candidate limit and reranking strategy from observed cases, then rerun the set after document or embedding changes. The useful output is a traceable decision about retrieval quality, not a claim that hybrid search is universally superior.
Find the stage that lost the evidence
Illustrative diagnostic cases; no benchmark scores are implied.
Scroll horizontally to read the table →
| Symptom | Inspect | Next experiment |
|---|---|---|
| Exact code not found | Indexing, tokenisation, filters | Exact identifier query |
| Paraphrase misses the manual | Semantic candidates and coverage | Compare lexical and semantic results |
| Relevant passage buried | Candidate order and duplicates | Evaluate reranking on the same cases |
| Good passage, wrong answer | Generation and conflicting context | Check claims against selected evidence |
Sources & further reading
Documentation consulted on .
FAQ / DECISIONS
Frequently asked questions
Is a vector database sufficient for RAG?+
It can support a first retrieval pipeline, but document extraction, permissions, ranking and answer evaluation still need to be designed. Test exact identifiers as well as natural-language questions.
Should every RAG system use a reranker?+
No. Add one when comparison on your own questions shows that better context selection justifies its extra latency and cost.