What’s Wrong With AI Eval “Science”?

AI evaluation produces many numbers. A new system is run on a collection of benchmarks, the scores go into a table, and models are compared. Consider:

Agent A scores 74% on benchmark B.

There are several ways this result can be misleading even when the number itself was calculated correctly. The tasks may represent only a narrow version of the work. The grader may reward an incorrect artifact. The score may change under another prompt or scaffold. Repeated runs may disagree. The evaluation environment may omit tools, permissions, or interactions present in routine use. Even a stable and correctly measured result may be inadequate for a decision if important failures are rare, costly, or difficult to detect. Treating these as one general benchmark-validity problem makes it harder to determine what should be tested next.

1. The work being evaluated may be underspecified

Agent evaluations often begin with labels such as “software engineering,” “research,” “tool use,” and “reasoning,” although those labels cover many different activities and settings. Two benchmarks can both evaluate research while requiring very different work: answering questions from a fixed corpus, finding sources on the web, producing a cited report, or maintaining a research workspace over many steps. Their scores therefore describe different activities. An evaluation needs to state what work is represented, under what conditions, what product the agent must produce, and which part of that product is evaluated. This is the problem addressed in our work on Designing Benchmarks for Knowledge Work [1].

2. The grader may be wrong

Passing the grader does not always mean the intended task was accomplished. Tests can miss incorrect outputs. Agents can exploit shortcuts. Parsers can alter answers. Environment bugs can make tasks impossible. A language-model grader can systematically disagree with the outcome a human reviewer would care about. Recent agent-benchmark audits show that task and reward implementation problems can materially change measured performance [2]. A precise score from a grader that accepts the wrong work is still a measurement failure. The relevant study is grader validation: define the intended outcome, construct known correct and incorrect artifacts, and measure false acceptance, false rejection, repeatability, and robustness.

3. The result may depend on evaluation choices

Agent systems are stochastic, but repeated seeds are only one source of variation. Prompt templates, scaffolds, tool versions, environment state, retry policies, task samples, and model versions can also change measured performance. Semantically similar prompts have been shown to change both absolute performance and model rankings [3]. The relevant study identifies which sources of variation should leave the result stable and varies those factors deliberately. Repeating one frozen configuration cannot answer this question.

4. The benchmark may differ from the workflow

A benchmark can be implemented correctly while omitting conditions that matter in routine work. An internal research agent may need to locate relevant information, work across permissioned systems, request clarification, preserve intermediate evidence, and leave a reviewable work product. A benchmark may instead provide a fixed corpus and grade only a final answer. These differences are separate from grader error or run-to-run instability. They require a direct comparison between the evaluation setting and the target workflow. Work on underspecification shows why strong held-out performance need not determine behavior under deployment-relevant conditions [5].

5. The average can hide the failure that matters

Two systems can have the same average success rate while behaving differently. One may fail consistently on a known subset of tasks. Another may fail unpredictably across repeated attempts. A third may have rare failures with much larger consequences. Recent work on agent reliability separates consistency, robustness, predictability, and safety instead of reducing reliability to one success rate [4]. For some uses, the distribution, severity, and detectability of failures matter more than the average.

6. A measured result does not define a decision

A system can clear a benchmark threshold while still being unsuitable for a particular workflow. The decision can depend on uncertainty, false-acceptance and false-rejection costs, human review burden, latency, alternatives, and the consequences of specific failures. These are decision-analysis questions. They should not be folded into the benchmark score itself.

Different failures require different studies

A problem with task representation requires a different study from a problem with the grader. Both differ from instability across runs or configurations. None establishes correspondence with another workflow. A well-measured workflow result still does not specify an acceptable deployment threshold. Agent evaluation therefore needs task and workflow specification, grader validation, sampling and uncertainty analysis, reproducibility testing, workflow correspondence, and explicit decision rules. The relevant methods already exist across measurement science, software testing, reliability engineering, survey methodology, clinical research, and causal inference. Agent evaluation needs to determine how they apply to the systems and workflows being studied.

References

1. Hua Y, Na H, Ayubcha C, Lian L. Designing Benchmarks for Knowledge Work. arXiv:2605.23262. 2026.
2. Zhu Y, Jin T, Pruksachatkun Y, et al. Establishing Best Practices for Building Rigorous Agentic Benchmarks. Advances in Neural Information Processing Systems. 2025.
3. Mizrahi M, Kaplan G, Malkin D, et al. State of What Art? A Call for Multi-Prompt LLM Evaluation. Transactions of the Association for Computational Linguistics. 2024;12:933–949.
4. Rabanser S, Kapoor S, Kirgis P, et al. Towards a Science of AI Agent Reliability. Proceedings of the 43rd International Conference on Machine Learning. 2026.
5. D'Amour A, Heller K, Moldovan D, et al. Underspecification Presents Challenges for Credibility in Modern Machine Learning. Journal of Machine Learning Research. 2022;23(226):1–61.