Separate fluency from correctness

An answer can read smoothly and still be wrong. Clear wording is useful, but it does not establish that a claim is supported. When working with generated answers, evaluation deserves a place in the design from the beginning.

Keep a small set of examples

Write down a few representative questions and what a useful answer should contain. Include questions that the available information cannot answer. These examples make it easier to notice when a change improves one case and breaks another.

Make evidence visible

For a document assistant, show the passage used to support an answer. Check that it actually supports the claim rather than merely mentioning the same topic.

A useful interface also has room for uncertainty. “There is not enough information in these documents” can be a better result than a confident guess.

Sample editorial content for this portfolio’s blog.