Evaluation6 min read
A practical checklist for evaluating chatbot answers
A repeatable review process for checking accuracy, relevance, clarity, and appropriate uncertainty.
Testing a chatbot with two or three easy questions is not enough. A small, repeatable evaluation set makes changes easier to compare.
Build a representative question set
- Include common questions that are clearly answered on the website.
- Add questions that require details from more than one passage.
- Test ambiguous wording and common spelling mistakes.
- Include questions the website does not answer.
Review four qualities
For each response, check factual support, relevance to the question, clarity of wording, and whether uncertainty is communicated appropriately. Record failures with the question, response, expected result, and source page so they can be reproduced.