Write down what good means
Start by describing useful behavior in the language of the task. A support assistant might need to find the correct policy, explain it clearly and avoid promising an exception it cannot authorize. Treat these as separate criteria. An answer can sound helpful while using the wrong source. Reviewers need concrete examples of acceptable and unacceptable responses so that their judgments mean roughly the same thing.
Build a representative set
Collect examples across ordinary requests, incomplete questions and cases outside the intended scope. Remove personal or confidential information unless its use is specifically authorized and appropriately controlled. Record the expected behavior alongside each example. Include situations where asking a follow-up question or declining to answer is the right outcome. Keep a portion of the set separate from day-to-day tuning to check whether improvements generalize beyond familiar examples.
Inspect failures, not just averages
An overall score can hide a small group of expensive mistakes. Review failures by type: missing evidence, incorrect interpretation, unauthorized disclosure, confusing language or inability to complete the task. Assign owners to the failures that matter and connect them to a release decision. A team can then explain which limitations it accepts, which need additional controls and which prevent deployment. This is more useful than a single unexplained percentage.
Maintain the evaluation
The workflow will change as policies, users and source documents change. Add relevant examples when a production issue exposes a gap, while keeping enough stable cases to compare releases. Record the configuration tested and the reason for each change. Schedule periodic human review rather than assuming automated checks cover every concern. The evaluation should become a maintained part of the product, with an owner and an explicit place in the release process.
Working through a similar question?
Tell us about your context and the decisions in front of you.
Talk to LUNAR LABS