Retrieval cannot repair an unclear knowledge estate
Chunking and embeddings matter, but they cannot resolve conflicting policies, missing ownership, inaccessible sources, or content that no longer reflects how the organization works.
A reliable retrieval system begins with source mapping: what information exists, who owns it, who may access it, how often it changes, and which source is authoritative when two documents disagree.
Permissions are part of relevance
A technically relevant result is not useful if the requesting person is not allowed to see it. Retrieval must carry identity and access context through indexing, filtering, ranking, generation, and source display.
Permission behavior should be tested directly, including changes in group membership and document access after indexing.
Freshness needs an operating model
Ingestion pipelines should expose failed sources, stale indexes, deleted content, and unexpected volume changes. Without those signals, teams may not know the assistant is using outdated context.
Ownership should be clear across source systems, the retrieval platform, and the product experience so stale or poor quality information has a defined resolution path.
Evaluate with real questions and evidence
A useful evaluation set reflects the questions people actually ask, the sources an expert would use, the expected access boundaries, and the cases where the system should abstain.
Measure retrieval and answer behavior separately. This helps the team identify whether a failure came from source coverage, ranking, context construction, or generation.
What evidence would your team need to operate this capability with confidence?