Build LLM Evals Around User Outcomes: Tasks, Graders, and Human Calibration
A practical eval guide for LLM apps and agents: representative tasks, outcome and trace checks, grader choice, repeated trials, and human review.
On this page

“Looks good in a few demos” is not an evaluation strategy. A useful eval makes a product expectation explicit, turns it into testable cases, and shows whether a change improved the outcomes users actually need without hiding new failures elsewhere.
Begin with a user outcome
Write one sentence that defines success for a real task. For a support assistant, it might be “answer a billing question from the current policy and cite the relevant clause, or hand off when the policy does not cover it.” For an agent that updates a record, success should include whether the correct record changed—not just whether the model said “done.”
Then define the system boundary. Does the measurement include retrieval, tool calls, retries, and downstream services, or only the final model response? If several system layers change together, an overall score can reveal a regression but not its cause. Track targeted component checks alongside end-to-end outcomes.
OpenAI’s evaluation guidance recommends task-specific datasets, realistic distributions, continuous evaluation, and human calibration for automated metrics. Anthropic’s agent-evaluation guide distinguishes the task, trial, grader, transcript, outcome, harness, and evaluation suite. These definitions help teams agree on what a score actually represents. [2] [1]
Build a dataset that resembles the work
Start with a small, versioned set of representative requests and known failure cases. Include common paths, boundary inputs, missing or contradictory information, adversarial prompts, and examples where the correct behavior is to decline or escalate. Add anonymized production examples only when privacy, consent, and retention rules permit; generated synthetic cases are useful for coverage but do not prove that the test distribution matches real use.
For each item, store the input, expected behavior, relevant context or environment, and the reason the case matters. Avoid baking one exact sentence into the expected output when many phrasings are valid. Conversely, do not use vague goals such as “be helpful” without a rubric that tells reviewers what counts as useful for this task.
A dataset should have clear ownership and version history. When the model, prompt, retrieval corpus, tools, or policy changes, record which version produced each run. Keep a held-out set for final comparison when teams are tuning against a known evaluation suite; repeatedly optimizing to the same examples can overfit the implementation to the test.
Match the grader to the claim
Use the simplest grader that can correctly measure the requirement:
- Code-based checks are fast and repeatable for schema validity, required fields, exact classification labels, successful tool invocation, or a database state change.
- Reference-based checks work when a task has a clear reference answer or known supporting facts. Similarity scores can flag differences, but a high text similarity score does not prove factual correctness.
- Model graders can assess nuanced properties such as whether a response follows a rubric or supports a recommendation with evidence. They are themselves models and can be inconsistent, biased, or sensitive to wording.
- Human review is slower but necessary for ambiguous criteria, high-impact outcomes, and calibration of automated judgments.
Anthropic’s evaluation guidance recommends selecting code, model, and human graders according to the task and notes the trade-off between objective checks and nuanced judgments. OpenAI’s grader documentation lists exact checks, similarity, model scoring, and code execution, but also displays a product lifecycle notice; teams using any hosted evaluation feature should verify its current status and retain portable copies of data and grading logic. [3] [4]
Evaluate agents at the outcome and trace level
A multi-step agent can produce a polished final answer after making a risky tool call, or it can claim success without changing the target system. Record the full execution trace: model turns, tool names and arguments, returned data, retries, intermediate decisions, and final environment state. Then grade multiple properties independently.
For a scheduling agent, separate at least these questions: Was the right calendar queried? Did the agent respect availability constraints? Was the intended meeting actually created? Did it disclose ambiguity or ask before sending invitations? A single “task success” score can hide that it reached the desired state through an unsafe or unauthorized path.
Anthropic’s agent-evaluation guide defines an outcome as the final environment state and a transcript as the complete record of a trial. It recommends repeated trials because model behavior varies between runs, and clean environments to avoid leftover state or resource flakiness. The engineering implication is to grade the side effect and the path, not only the generated explanation. [1]
Measure variation and keep scores interpretable
Repeat tasks when nondeterminism can change the result. Report a success rate across trials, not just the best run. Break results down by task family, user segment, language, or input difficulty when those slices matter and sample sizes support a meaningful comparison. Track cost and latency with quality so an apparent improvement is not simply an unbounded increase in retries or model calls.
Avoid a single composite score unless the weights are justified by user impact. A system that improves tone but doubles incorrect answers should not appear better because the scores average out. For safety or policy checks, make critical failures visible as separate gates; do not let strong performance on easy cases mask a serious failure mode.
Model graders need their own evaluation. Compare their judgments with expert labels on a representative sample; check for position bias in pairwise comparisons, sensitivity to output length, and instability across grader versions. Use clear rubrics and require evidence for factual judgments. If two human experts cannot agree on a case, an automated grader should not be treated as an objective oracle.
Turn evals into a development loop
Run a fast, narrow suite during development and a broader suite before release. When an evaluation fails, inspect the input, trace, grader output, and environment before changing a prompt. Add a confirmed failure to the regression set so the fix remains visible. After deployment, production monitoring and user feedback should add new cases; offline evals cannot represent every real interaction.
OpenAI’s guidance describes an iterative cycle: define the objective, collect a dataset, choose metrics, compare versions, and continue evaluating as the application changes. Anthropic describes eval suites as living artifacts that need maintenance. Neither a test score nor an automated grader proves that an application is safe or fair in every context. [2] [1]
For agent systems, add a versioned snapshot of tools, permissions, model, prompts, environment, and data. For model-only workflows, store the prompt, model version, decoding settings, and evaluator. Pinning every variable may not always be possible, but recording what changed helps distinguish model regressions from dataset, retrieval, or tool changes.
A practical first eval
- Choose one consequential user task and define a pass condition.
- Collect typical, edge, and known failure examples with expected behavior.
- Add exact checks for objective requirements and a human-reviewed rubric for nuanced ones.
- Run the system in a clean, versioned environment and save traces plus final outcomes.
- Repeat variable tasks and report pass rate, failures, latency, and cost by relevant slice.
- Compare against a baseline, inspect disagreements, and add confirmed failures to regression coverage.
- Calibrate model graders against human judgments and revise criteria when reviewers disagree.
For an evaluation of a prompt or model change, the question is not “Which version sounds better?” It is “Which version more consistently completes the defined task, within its limits, for the users and conditions we care about?” A useful eval makes that comparison reproducible and keeps the failure cases visible.
Frequently asked questions
What should an LLM evaluation measure?
Measure a task-specific success criterion that reflects the application, such as correct classification, grounded answers, a valid state change, or safe escalation. For agents, inspect both the final outcome and the steps that produced it.
Should every LLM eval use an LLM judge?
No. Use exact code checks where requirements are objective, human review for high-stakes or ambiguous judgments, and model graders only where nuanced scoring is valuable and calibrated against human judgments.
How many times should I run a stochastic agent task?
One run can hide variability. Repeat tasks when output variation matters, report the pass rate and uncertainty, and separate task difficulty from environmental flakiness. The number of trials depends on risk and the precision needed.



/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.