Which questions should the assistant answer?
Start with the questions customers ask. Support tickets, chat logs and emails will show you the most frequent ones. For each, check whether it can be answered from company information that someone maintains: delivery terms, opening hours, product instructions, the returns policy. If the answer exists only in an experienced employee’s head, the AI chatbot has nothing reliable to draw on.
Give each type of information a responsible person: prices, policies, availability, product instructions. When a policy changes, someone has to update the source, otherwise the assistant will keep giving the old answer with complete confidence.
Then mark the cases that need a person from the start: a dispute, a complaint, an unclear request about a specific account, anything that involves an exception to the rules. The assistant’s scope should match the information that is accessible and approved. Promising customers an assistant that “answers everything” creates expectations it cannot meet, and every wrong answer costs more trust than an honest handover.
How should the handover to a person work?
Treat the handover as part of the answer, not as a failure. The assistant needs a defined way to recognise that it cannot give a supported answer, because the question falls outside its scope, the sources do not cover it or the customer is clearly unhappy, and a defined way to act on that.
When it hands over, the customer should understand what happens next and what information is needed, without having to repeat everything to the person who picks up the case. Where an integration exists, pass a short summary to the right queue: what the customer asked, what the assistant has already answered and which details have been collected. Few things annoy customers more than explaining the same problem twice. If customers message you on Telegram, a Telegram AI bot can follow the same handover rules inside the chat.
Keep the original question and the source references available to the person reviewing the case, with suitable access controls. The support team can then see not only what the customer wanted but also what the assistant based its reply on, which is essential when you need to work out why an answer went wrong.
How do you test the answers and the whole service?
Build a review set that includes straightforward questions, ambiguous ones and questions the assistant should not answer at all. Then check accuracy against the sources, the tone and whether escalation happened when it should have. Fluency on its own is a poor measure: an assistant can write beautifully and still be wrong.
Measure the service as a whole. What share of conversations ended with the customer’s issue resolved, and how much effort did reviewing and correcting the assistant take? Define both terms explicitly before you start counting. “Resolved” means different things to different teams, and without a written definition the figures cannot be compared over time.
Repeat the checks after any change to your knowledge base or products. The same rule applies to an AI knowledge assistant for your staff. A smooth answer can still be outdated or wrong, so the assistant needs ongoing monitoring and a person who can correct the underlying information rather than the individual reply. If the source is wrong, fixing one conversation will not stop the same mistake tomorrow.
Checklist: support assistant
- List the questions the assistant may answer and the person responsible for each source of information.
- Define what the assistant does when it has no supported answer and when it must escalate.
- Test the transfer to the support queue and the context the reviewer receives.
- Measure accurate resolution and review effort using definitions written down in advance.
Example: delivery questions and a disputed order
A customer asks about delivery options, and the assistant answers from an approved reference document. Later the same customer asks about an order they are disputing. Here the assistant does not try to settle the dispute itself: it collects the minimum information the team has specified, explains what happens next and passes the case to support.
The example tests two things at once: whether the factual answer about delivery matches the source, and whether the disputed case reaches the right queue with enough context for the person who handles it.
