What’s Wrong With AI Eval “Science”?

AI evaluation produces many numbers. A new system is run on a collection of benchmarks, the scores go into a table, models are ranked, and the result becomes evidence that one system is better, more capable, safer, or more ready for use. It is often unclear what exactly the number describes. Consider a statement such as:

Agent A scores 74% on benchmark B.

To reproduce or interpret that result, we may need to know the benchmark version, task subset, model version, scaffold, system prompt, tools, permissions, time and token budgets, retry policy, environment state, grader, parser, and number of runs. Change some of these and the score changes [3]. Yet the result is frequently discussed as though 74% were a stable property of Agent A. That is the first scientific problem.

1. The thing being measured is often unclear

Scientific measurement starts by specifying the object and conditions of measurement. Agent evaluations often begin with labels such as “software engineering,” “research,” “tool use,” and “reasoning,” although those labels cover many different activities and settings. Two benchmarks can both claim to evaluate research while requiring very different work: answering questions from a fixed corpus, finding sources on the web, producing a cited report, or maintaining a research workspace over many steps. The resulting scores do not mean the same thing. Before asking whether an agent performs well, an evaluation needs to state what work is represented, under what conditions, what product the agent must produce, and what part of that product is actually evaluated. This is the problem addressed in our work on Designing Benchmarks for Knowledge Work [1].

2. The measuring instrument is part of the result

An agent benchmark includes a set of tasks and the apparatus used to run and score them. The scaffold determines how the model receives information and acts, tools determine what actions are possible, the environment determines what state exists, and tests and graders determine what counts as success. These components can change measured performance substantially. The system being evaluated and the apparatus used to evaluate it are tightly coupled. A result therefore needs a system identity. “Claude scored X” or “GPT scored Y” is usually incomplete. The system identity can be reported as:

model + scaffold + tools + environment + evaluation protocol + version.

Without this information, comparisons can mix changes in the model with changes in the measurement process.

3. A successful run may still be a measurement failure

Passing the grader does not always mean the intended task was accomplished. Tests can miss incorrect outputs, agents can exploit shortcuts, parsers can alter answers, environment bugs can make tasks impossible, and a language-model grader can disagree systematically with the outcome a human reviewer would care about. Software testing, laboratory science, and diagnostic-test research would treat these as instrument-validation problems. Agent evaluation often treats them as benchmark implementation details. Recent agent-benchmark audits show that grader and task implementation problems can materially change measured performance [2]. That distinction matters because a highly precise score from a faulty grader is still a faulty measurement.

4. A score from one evaluation does not automatically travel

Suppose an agent performs well on a carefully constructed benchmark. The next claim may concern materially different conditions (e.g., another task distribution, private company data, or different tools and permissions).

Performance in the benchmark setting is evidence about the benchmark setting. Using it to support a claim about another setting requires additional evidence about the relationship between the two. Clinical research distinguishes controlled efficacy from effectiveness in routine practice. Causal inference studies transportability. Survey methodology distinguishes a sampled frame from the population one wants to describe. Agent evaluation has the same problem, with fewer established methods for addressing it. Related work on underspecification shows why strong held-out performance need not determine deployment behavior [5].

5. The mean can hide the behaviour that matters

Agent systems are stochastic. An agent that succeeds on nine out of ten tasks can behave in several very different ways: it may reliably fail on one particular class of task, behave inconsistently on almost every task, or perform well most of the time and occasionally produce a severe failure. Those systems can have the same average score. For some scientific questions, averaging across runs is appropriate. For a deployment decision, the distribution and structure of failures may matter as much as the mean. Reliability therefore has to remain part of the evidence instead of disappearing during aggregation [4].

6. Evaluation results become claims and decisions

Evaluation results are also used to support broader claims and decisions. A measured result such as 74% under configuration C can become competitive with expert performance, then reliable enough for workflow X, and eventually ready to deploy. Each statement requires additional evidence [6]. The transitions are often implicit. AES therefore separates the evaluation result from the claim made with it. A related note asks:

What Does an Agent Evaluation Actually Support?

The note follows the evidence from observed runs through scoring, generalization, target-setting extrapolation, and finally a decision, and defines the claim ceiling as the strongest conclusion supported by the current evidence. A companion note focuses on one of those transitions:

How Close Is an Agent Evaluation to the Real Workflow?

It compares benchmark conditions with the workflow the result is eventually supposed to describe.

What would make agent evaluation more scientific?

Agent evaluation needs clearer measurement objects, validated graders, explicit sampling frames, repeated and reproducible measurements, preserved uncertainty, evidence for transfer between settings, and a visible separation between a result and the decision made from it. Most of these problems are old. Measurement science, clinical research, software testing, reliability engineering, survey methodology, causal inference, and evidence synthesis already have methods for dealing with them. AES studies how these methods can be adapted to agents, tested, and incorporated into reporting and review infrastructure.

References

1. Hua Y, Na H, Ayubcha C, Lian L. Designing Benchmarks for Knowledge Work. arXiv:2605.23262. 2026.
2. Zhu Y, Jin T, Pruksachatkun Y, et al. Establishing Best Practices for Building Rigorous Agentic Benchmarks. NeurIPS. 2025.
3. Mizrahi M, Kaplan G, Malkin D, et al. State of What Art? A Call for Multi-Prompt LLM Evaluation. Transactions of the Association for Computational Linguistics. 2024;12:933–949.
4. Rabanser S, Kapoor S, Kirgis P, et al. Towards a Science of AI Agent Reliability. ICML. 2026.
5. D'Amour A, Heller K, Moldovan D, et al. Underspecification Presents Challenges for Credibility in Modern Machine Learning. Journal of Machine Learning Research. 2022;23(226):1–61.
6. Kane MT. Validating the Interpretations and Uses of Test Scores. Journal of Educational Measurement. 2013;50(1):1–73.