Professional domain 4 · Evaluation, testing and optimisation
- Success criteria are specific, measurable, achievable and relevant. Cover accuracy, latency, cost, safety and security, not accuracy alone.
- Evaluation sets mirror the real mix of inputs and include edge cases. Anthropic's advice: more test cases with automated grading beat a few hand-graded ones. Keep a held-out set, and add real production failures as you find them.
| Grading method | Strength | Watch out for |
|---|---|---|
| Code-based (exact or string match) | Fastest, most reliable, scales easily | No nuance for open-ended answers |
| LLM-based | Fast, flexible, handles judgment | Needs a clear rubric and scale, reasoning before the verdict, and a reliability check before you scale it |
| Human | Most flexible, highest quality | Slow and expensive; use it to calibrate the others |
- Mixed methods: offline evaluation before release, A/B tests in production, and human review of samples.
- A/B tests: change one thing, decide the metric and sample size in advance, and compare on the same kind of traffic.
Diagnosing problems
| Symptom | Likely cause | First fix |
|---|---|---|
| Consistently wrong on one kind of input | Prompt failure | Clearer instructions, examples of that case |
| Confident claims the sources don't support | Hallucination | Allow "I don't know", quote sources first, require citations, restrict answers to the provided documents |
| The right answer was never in the context | Retrieval failure | Fix chunking, search or reranking before the prompt |
| Fails on hard cases, or is slow and costly on easy ones | Model mismatch | Move up or down a model, or change the effort |
- Optimising: cache stable prefixes, batch non-urgent work, use a smaller model or lower effort for easy steps, trim context and outputs, stream for responsiveness. Check the evaluation set after each change.
- Monitoring: log and trace in production, watch dashboards and alerts, review samples, and feed what you find back into the evaluation set.
See also: Professional domain 3 · Accuracy, latency and cost, Professional domain 2 · Prompt reuse: caching, modular prompts and Skills and Professional domain 3 · Observability at scale.
Docs: Success criteria and evaluationsReduce hallucinationsReducing latencyPrompt caching
Practise Domain 4