Evaluating business LLM applications: a practical checklist
How to measure whether an AI assistant is actually working: building a test set, choosing metrics and catching regressions before users do.
Evaluate an AI assistant by building a test set of 50 to 200 real questions with expert-approved answers, scoring correctness, groundedness, completeness, refusal quality, latency and cost on every change, sampling real conversations weekly for human review, and monitoring production for refusals, low-confidence answers and cost spikes.
Most AI demos look impressive because someone chose the questions. Real users ask things you did not expect. The only way to know whether an assistant works is to measure it on real tasks, repeatedly. This checklist is the process we use.
1. How do you build the test set?
- Collect 50 to 200 real questions from the people who will use the assistant: support tickets, emails, chat logs.
- For each, write the correct answer or the source document that contains it. Ask a subject-matter expert, not the developer.
- Include hard cases: ambiguous questions, questions with no answer, and questions that should be refused.
2. Decide what “good” means
| Metric | Question it answers |
|---|---|
| Correctness | Is the answer factually right? |
| Groundedness | Is every claim supported by a cited source? |
| Completeness | Does it include what a good answer must? |
| Refusal quality | Does it say “I don’t know” when it should? |
| Latency and cost | Is it fast and affordable at your volume? |
3. Score every change
Run the full test set after every change to prompts, models, retrieval settings or documents. Track the scores over time. A change that helps one question type often hurts another, and only the numbers will show it.
4. Keep a human review loop
Automated scoring, including model-based judging, is useful but imperfect. Sample real conversations weekly, have a person grade them, and feed the failures back into the test set.
5. Monitor in production
- Log questions, retrieved sources, answers and user feedback.
- Alert on spikes in refusals, low-confidence answers or thumbs-down ratings.
- Review cost per conversation so a prompt change cannot quietly double the bill.
For the choice of approach in the first place, see RAG vs fine-tuning vs prompting.