In this article
The same question must not imply the same answer
Consider an employee and a manager asking about compensation policy. Their permitted sources may differ. Build the retrieval filter from the authenticated identity and current memberships, not a role mentioned in the question. If the manager’s answer is cached, the employee must not inherit it through a cache key that only contains the question text.
Test the whole response surface: passages, generated statements, document titles, citations and suggested follow-up questions. Even a source title can disclose information. A citation link should enforce access again when opened; an answer generated yesterday does not grant access today. Revoking membership must affect subsequent retrieval and any cached responses whose authorisation context is no longer valid.
Apply authorization before retrieval
Indexing content does not remove its access rules. Carry user, tenant and document policy into retrieval so forbidden chunks cannot enter the model context. Filtering only the final answer is too late.
Preserve source identity, version, owner and effective permissions through ingestion. Treat shared links, deleted documents and permission changes as lifecycle events that update or remove indexed material.
Make provenance visible and inspectable
Return citations that resolve to the exact authorised source and enough context for a reader to verify the statement. A list of document titles is not useful when it cannot show which evidence supported which conclusion.
Separate “no relevant evidence” from provider failure and from an answer that conflicts with sources. The safe response may be to abstain and route the question rather than generate plausible filler.
Evidence has three obligations
Useful answers must also respect access and remain inspectable.
Allowed
The user has access before retrieval selects the passage.
Traceable
The answer points to the document and the relevant version.
Evaluated
Retrieval and generated claims are checked separately.
Evaluate the retrieval and answer separately
Build a representative question set with expected sources, permission cases, time-sensitive content, ambiguous wording and unanswerable requests. Measure whether retrieval found the evidence before judging answer wording.
Track citation support, missing evidence, unsafe access, abstention and latency by question class. Re-run the set after chunking, embedding, reranking, model or corpus changes and keep a rollback path.
Check the claim, not the presence of a citation
A response can contain a real link and still misrepresent the source. Ask whether the cited passage supports the exact claim, including scope, date and exceptions. For a travel-policy answer, a hotel allowance for one country is not evidence for another. Preserve document revision and passage identifiers so reviewers can inspect the material used at generation time.
Include questions with no answer, conflicting policies and revoked documents in the evaluation set. The expected outcome may be clarification, refusal or a request for a policy owner to resolve the conflict. Separate retrieval failures from unsupported generation: improving the prompt cannot recover a document that the search stage never returned. Report these categories to the people who can actually fix them.
Treat ingestion changes like application releases
Replacing an index, changing chunk boundaries or updating an access label can alter answers without any application-code change. Version the ingestion configuration and retain a way to trace which document set produced a response. Run representative questions after an import, including negative authorisation tests, before considering the new index ready.
A rollback to an older index needs special care: it must not restore access to documents that were deleted or revoked. Keep current access policy authoritative even during recovery. Give source owners an observable update path, with failed imports and stale revisions visible. That turns RAG into a maintained knowledge service rather than a one-time document upload whose reliability slowly drifts.
FAQ / DECISIONS
Frequently asked questions
Does adding citations eliminate hallucinations?+
No. A citation can be irrelevant or fail to support the claim. Evaluation must check entailment, retrieval completeness and permission safety.