Evaluation Science Brief
How Do You Know Your Agent Grader Works?
An agent submits an incorrect patch, and the tests pass. The score now counts a failure as a success. Agent evaluation usually treats the grader as part of the benchmark code and rarely checks it. Fields that depend on measurement treat the equivalent tool as an instrument that has to be validated before use [1,2]. The same check applies to agent graders.
Why grader error matters more than it seems
Suppose a grader is 95% sensitive and 90% specific, while only 20% of attempts on a hard benchmark actually succeed. Among 1,000 attempts, 190 of the 200 successes would be accepted, but so would 80 of the 800 failures. The reported pass rate would be 27%, and about 30% of the reported successes would be false. Grader error also changes comparisons. If the grader is equally lenient with two agents, the gap between them shrinks. If it is more lenient with one of them, the ranking can reverse.
Step 1: State what success means
Write down what the grader is supposed to recognize. For a coding task, the tests passing is the implemented check, while the intended outcome is closer to "the requested change works and nothing else broke." Then choose a reference standard to judge outputs independently of the grader, usually review by an expert who knows the task. If experts disagree on some cases, record how often, so that the grader does not end up defining success for itself.
Step 2: Test the grader on outputs with known answers
Software testing checks a test suite by planting deliberate faults and counting how many it catches, which is called mutation testing [2]. For an agent grader, build a small challenge set with two kinds of outputs. The first kind is wrong but plausible, for example a patch with a subtle off-by-one error. The second kind is correct but unusual, for example a valid solution written in a different style. A few dozen of each is enough to see whether the grader accepts wrong work or rejects right work.
Step 3: Check the grader on each agent separately
Because grader error can differ between agents, the challenge set is not enough on its own. Take a sample of each agent's real outputs (for example, 50 runs per agent), have an expert label them, and estimate the grader's errors for each agent. If the errors differ, correct each agent's score with its own error rates before comparing them.
Step 4: Check that the grader is consistent
Agreement with human ratings is often summarized as a correlation, but correlation measures association, and a judge can rank outputs like a human while being systematically more lenient [3]. Report agreement with the confusion matrix instead. For an LLM judge, also re-grade the same outputs several times and after harmless changes such as reformatting, and treat any variation as part of the score's uncertainty [1]. If the tests, rubric, or judge model change, repeat the validation.
Report the grader's error with the score
For example:
LLM judge, validated on 50 expert-labeled runs per agent: false acceptance 8% for Agent A and 19% for Agent B. Corrected success rates: 61% and 52%.
| Field | Record |
|---|---|
| Target outcome | What counts as task success? |
| Reference standard | Who or what judged the outputs independently? |
| Challenge set | Wrong-but-plausible and correct-but-unusual outputs |
| Errors by agent | False acceptance and false rejection for each agent |
| Consistency | Re-grading and harmless changes |
| Version | Grader, rubric or tests, judge model, date |
References
- International Council for Harmonisation. ICH Q2(R2): Validation of Analytical Procedures. 2023.
- Papadakis M, Kintis M, Zhang J, Jia Y, Le Traon Y, Harman M. Mutation Testing Advances: An Analysis and Survey. Advances in Computers. 2019;112:275–378.
- Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement90837-8). Lancet. 1986;1(8476):307–310.