How Do You Know Your Agent Grader Works?

An agent submits an incorrect patch. The tests pass. Was that an agent success, or a grader failure? Agent evaluation often treats the grader as part of the benchmark implementation. Other measurement-heavy fields treat the analogous object as an instrument that must itself be validated. Analytical procedure validation asks whether a method is fit for its intended purpose, including its accuracy, precision, specificity, and robustness [1]. Software testing asks whether a test suite actually detects faults [2]. Diagnostic-test research separates the accuracy of the test from the prevalence of the condition it is used to detect. Agent evaluation requires the same separation between grader accuracy and the prevalence of genuine task success.

Start with the target outcome

Before testing a grader, state what it is supposed to recognize. For a coding task, “the test suite passes” may be the implemented criterion, while the target outcome is closer to “the requested change is correctly implemented without introducing material regressions.” For a research task, the implemented criterion might be a rubric score, while the target outcome concerns whether the analysis is substantively correct and reviewable. The validation record should therefore name:

Target outcome: what genuine task success means.
Reference standard: how challenge cases will be classified independently of the grader being tested.

A reference standard can itself be imperfect. Its identity and uncertainty should still be explicit so the grader does not define success circularly.

Inject known mistakes

Mutation testing evaluates a test suite by injecting deliberate faults and measuring which faults the suite detects [2]. The same idea transfers directly to agent graders. Construct a challenge set containing two types of artifacts:

Wrong but plausible outputs. These should contain realistic faults the grader is supposed to catch: an incorrect calculation, a missing required change, an unsupported claim, a corrupted file, a superficially correct answer produced from the wrong state, or another task-specific failure.
Correct but unusual outputs. These should satisfy the task while varying irrelevant features such as formatting, ordering, wording, implementation style, or file structure.

Then report two quantities:

False acceptance: incorrect work accepted as successful.
False rejection: correct work rejected as unsuccessful.

This tests the grader against known cases before the grader is used to make claims about unknown cases.

Treat grading as diagnostic classification

If genuine task success can be established on a challenge set, the grader can be described using the same quantities used for a diagnostic test: sensitivity to true success and specificity to true failure. The base rate also matters. Suppose a grader is 95% sensitive and 90% specific, while only 20% of attempts on a hard benchmark are genuinely successful. Among 1,000 attempts, 190 genuine successes would be accepted, but 80 failures would also be accepted. The reported pass rate would be 27%, and about 30% of the reported successes would be false positives. These numbers are an example, not an empirical finding about any current benchmark; under these assumptions, one agreement statistic would not summarize the grader's error characteristics.

Agreement is not correlation

A common validation exercise compares an automated judge with human ratings and reports a high correlation. Bland and Altman's classic argument applies directly: correlation measures association, not agreement [3]. Two methods can rank outputs similarly while one is systematically more lenient or more severe. For continuous scores, inspect bias and the distribution of differences using an agreement method appropriate to the design. For categorical decisions, report an appropriate agreement statistic together with the confusion matrix. For an LLM judge, repeatability should also be tested. Re-grade the same artifacts under the same configuration, then deliberately vary small features that should not change the judgement. Analytical method validation calls the latter robustness: performance under small deliberate variations in method parameters [1]. Perturbations for an agent grader can include harmless changes in formatting, order, whitespace, file names, or wording. If these change the judgement materially, that variation belongs in the evaluation uncertainty.

Version the grader

A grader can change without the benchmark tasks changing: the test suite, parser, rubric prompt, LLM judge model, or a dependency that affects execution can change. The grader identity should therefore include its version, prompt or rubric, parser, tests, relevant dependencies, and date. If the grader changes, prior validation evidence may not transfer automatically.

Grader Validation Card

If these fields are unknown, the evaluation contains unmeasured grader uncertainty.

References

1. International Council for Harmonisation. ICH Q2(R2): Validation of Analytical Procedures. 2023.
2. Papadakis M, Kintis M, Zhang J, Jia Y, Le Traon Y, Harman M. Mutation Testing Advances: An Analysis and Survey. Advances in Computers. 2019;112:275–378. doi:10.1016/bs.adcom.2018.03.015.
3. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307–310. PMID:2868172.
4. Zhu Y, Jin T, Pruksachatkun Y, et al. Establishing Best Practices for Building Rigorous Agentic Benchmarks. NeurIPS. 2025.
5. Bossuyt PM, Reitsma JB, Bruns DE, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527.