Agent Evaluation Requires Methods Beyond Psychometrics

Construct validity has become an unusually broad and popular diagnosis in AI evaluation.[1,2] A benchmark does not resemble the target work closely enough: construct validity. The grader rewards the wrong output: construct validity. Performance on the benchmark does not transfer to deployment: again, construct validity. These are related problems, but they are not the same problem, and treating them as one does not help us design better evaluations.

There is a good reason psychometric language has become common in AI evaluation. Many benchmarks look like tests: we define a capability, construct tasks intended to elicit it, observe responses, score those responses, and interpret the resulting performance. Questions about what is being measured, how scores behave, and what interpretations they support clearly matter. Construct validity was developed around this relationship between observed test performance and the construct we want to interpret from it,[4] while later work in psychometrics substantially broadened validity beyond this narrow formulation. Messick incorporated generalization, external evidence, and consequences into a unified account of validity,[5] and Kane made the inferential structure explicit by asking what assumptions are required to move from observed performance to an intended interpretation or use.[6]

Psychometrics is relevant to evaluation, but agent evaluation is often doing more than measurement, and calling every weakness a construct-validity problem can obscure what study is actually needed.

Consider a coding benchmark whose tasks closely resemble the work it claims to evaluate: the repositories are realistic, the issues require meaningful code changes, and successful completion corresponds reasonably well to the work activity of interest.[3] Even then, the evaluation can fail for reasons that have little to do with whether the construct was well represented. The grader may accept patches containing substantive errors; the sampled repositories may cover only a narrow part of the target task distribution; repeated runs may produce substantially different outcomes; or two systems may be compared under different scaffolds or tool access. Performance on public repositories may also say little about performance in a company's internal codebase, and even an accurately estimated success rate does not by itself determine whether autonomous deployment is acceptable.

These failures occur at different inferential steps:

A problem with task representation is different from a problem with the grader. Both are different from uncertainty due to task sampling or stochastic execution, and none of them establishes whether performance will carry to another environment or support a deployment decision.

For a more detailed discussion of the evidence required across these steps, see What Should an Agent Evaluation Report So Someone Else Can Use It?. Using one construct-validity label for all of them hides differences that change how an evaluation should be designed and interpreted.

For agents, the object being evaluated is often not well represented by the usual image of a respondent answering a collection of independent items. An agent may include a model, scaffold, tools, persistent state, an environment that changes in response to its actions, and other actors that provide information or feedback. Later opportunities can depend on earlier choices, and changing one component can alter the behavior of the whole system.

In such settings, the quantity of interest is not always a latent capability score. We may want to estimate the probability that a particular system configuration completes a workflow, how frequently a specific failure occurs, whether changing one component causes performance to improve, or whether behavior remains acceptable when the task distribution or operating environment changes. These questions can all be evaluated with benchmark-like tasks and can all produce numerical scores, but they are not therefore the same kind of scientific problem.

If the question is causal, the comparison has to support a causal interpretation. If the concern is an operational failure, the evaluation has to expose the system to conditions under which that failure could occur. If the result is intended to support a decision, the relevant errors and acceptable level of performance depend on that decision.

Modern psychometrics itself already recognizes that validation extends beyond a narrow question of construct representation. Messick's framework explicitly included generalization, external relationships, and consequences,[5] while Kane's argument-based approach begins with the interpretations and uses that observed scores are expected to support.[6] More recent work on AI evaluation has likewise separated issues such as robustness and generalization from success on the original benchmark.[7]

The analysis can start with the inference the evaluation is supposed to support. What was directly observed? What conclusion is being drawn from it? What has to be true for that conclusion to hold, and where could that step fail? Sometimes the answer is fundamentally about measurement and construct validity. In other cases, the main issue is the system, comparison, environment, or decision being studied.

Agent evaluation needs methods for each inferential problem. Construct validity remains important, but it should not become a catch-all label for problems in scoring, generalization, extrapolation, or decisions.

References

1. Bean AM, Kearns RO, Romanou A, et al. Measuring What Matters: Construct Validity in Large Language Model Benchmarks. Advances in Neural Information Processing Systems. 2025.
2. Alaa A, Hartvigsen T, Golchini N, et al. Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity. Proceedings of the 42nd International Conference on Machine Learning. Proceedings of Machine Learning Research. 2025;267:80991–81004.
3. Hua Y, Na H, Ayubcha C, Lian L. Design and Report Benchmarks for Knowledge Work. arXiv. 2026. arXiv:2605.23262. doi:10.48550/arXiv.2605.23262.
4. Cronbach LJ, Meehl PE. Construct validity in psychological tests. Psychological Bulletin. 1955;52(4):281–302. doi:10.1037/h0040957.
5. Messick S. Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist. 1995;50(9):741–749. doi:10.1037/0003-066X.50.9.741.
6. Kane MT. Validating the interpretations and uses of test scores. Journal of Educational Measurement. 2013;50(1):1–73. doi:10.1111/jedm.12000.
7. Mitchell M. Six principles for evaluating cognitive capabilities in AI models. AI Magazine. 2026;47(2):e70061. doi:10.1002/aaai.70061.