Building a Support Assistant That Does Not Invent Answers

A support assistant that guesses confidently creates more work than it removes. The design goal is not maximum answer rate; it is maximum trustworthiness.

1. Collect what the bot may say

Index your help centre, product documentation and release notes. Do not index sales material or anything aspirational — the bot will repeat it as fact.

2. Retrieve before generating

Set a similarity threshold. Below it, do not generate an answer at all. This single rule prevents most fabrication and is the most important design decision in the system.

3. Instruct the refusal explicitly

Tell the model that when the retrieved passages do not contain the answer, it must say so and offer to hand the conversation to a person. Vague instructions like "be helpful" push it toward invention.

4. Show the source

Attach the article link to each answer. Users self-serve faster when they can verify, and your support team can see exactly which document produced a bad answer.

5. Log everything

• Question text and whether retrieval succeeded.
• Which passages were retrieved and their similarity scores.
• Whether the user accepted the answer, rephrased, or asked for a human.

6. Review weekly at first

Low-similarity questions are your content backlog. Repeated escalations on the same topic usually mean the source document is unclear rather than that the model is weak.