Build an internal knowledge assistant with usable evidence
Choose approved documents, preserve permissions and test citations, missing answers and conflicting information in a knowledge assistant.
An internal knowledge assistant should help staff find and interpret approved information. Its usefulness depends on which documents it can access, how current those documents are and whether users can inspect the evidence behind an answer. A conversational interface does not repair an outdated or contradictory knowledge base by itself.
Define the knowledge boundary
Start with a bounded collection, such as current service procedures for one team. Identify document owners, effective dates and superseded versions. Remove duplicates or mark which version governs. When two documents disagree, record a resolution rule or return the conflict for review rather than asking the model to choose whichever wording sounds more convincing.
Preserve access permissions when retrieving information. A user should not gain access to a restricted document simply because it has been indexed for search. Microsoft’s document-level access guidance describes permission-aware retrieval options; the chosen implementation must also authenticate the requester and test that prohibited content is excluded before answer generation.
Require answers that can be checked
Return source references with enough detail for the user to verify the relevant passage. A citation should support the nearby claim, not merely point to a broadly related document. Ask the assistant to distinguish direct instructions from interpretation and to say when the available material does not answer the question.
Treat retrieved text as evidence, not as authority to change the assistant’s operating rules. A document can contain instructions intended for people, examples or malicious content. Restrict tool permissions independently of the text being retrieved, and keep actions such as sending messages or modifying records outside an initial question-answering pilot.
Evaluate retrieval and answering separately
Build questions with known supporting passages, including paraphrases, outdated terms and questions that should receive no answer. Check whether retrieval finds the right evidence before evaluating the generated response. If the relevant document never reaches the model, changing the writing prompt may improve fluency without correcting the underlying failure.
Include permission tests using different user roles, and repeat them when documents move or access changes. Measure unsupported claims, incorrect citations and stale answers rather than only user satisfaction. Keep a visible way to report a bad answer and direct the report to the document owner or system maintainer responsible for fixing the cause.
Practical checklist
- Choose a bounded, owned and current document collection.
- Enforce document access before generating an answer.
- Test citations against the specific claims they support.
- Include unanswerable, conflicting and restricted questions.
Illustrative setup: service policy questions
A coordinator asks whether a service includes weekend support. The assistant finds an old proposal and a current policy with different wording. The pilot should surface the conflict and point to the current policy owner, rather than combining the documents into a new promise. The coordinator can verify the source before replying to a customer.
Common questions
Does adding more documents improve answers?
Not automatically. More material can introduce duplicates, outdated policies and irrelevant passages. Add collections deliberately and evaluate their effect on the questions users actually ask.
Can citations guarantee an accurate answer?
No. A source link can be irrelevant or misinterpreted. Evaluation should check whether the cited passage supports each important claim and whether the user is allowed to access it.
Further reading
Start with your actual workflow.
Turn the useful parts of this guide into a focused project brief.
Shape your project