What Should an Agent Evaluation Report So Someone Else Can Use It?

A table can say:

Agent A — 74%

and still leave most of the evaluation unrecoverable. Was this the base model or a particular scaffold? Which task version? How were crashes and retries handled? What did the grader inspect? Someone who wants to use the result has to reconstruct the experiment from a paper, a repository, and a private script, and often cannot.

Step 1: Put the setup next to the score

Report the system (model, scaffold, tools, permissions, date), the task set and how its tasks were selected, and the run policy. Clinical trials treat events such as dropout as part of what is being estimated, because different handling rules change the result [1]. Agent runs have the same problem with timeouts, crashes, and partial completion, so state how each was counted. Our brief on what an evaluation supports gives a template for these fields.

Step 2: Keep the evidence behind the number

Report the number of tasks and runs, the uncertainty, how the grader was validated, and the main types of failure. When grader error or exploitation of the environment is plausible, keep the trajectories as well. For a private evaluation, state what was kept and who reviewed it.

Step 3: State the conclusion and its limits

Say which conclusion the result is meant to support and which assumptions have not been tested [2,3]. Measurement to Meaning applies this approach to AI evaluation [4]. If the evaluation covers one configuration, say so. If its relevance to another workflow has not been tested, say what comparison is missing.

What a usable report looks like

Agent A (model M, scaffold S, evaluated September 2026) solved 74% of 300 tasks from public Python repositories (95% interval 69–79%). Timeouts were counted as failures. The grader accepted 6% of wrong patches in a check on 50 expert-labeled runs. The result has not been compared with work in our own codebase.

References

1. International Council for Harmonisation. ICH E9(R1): Addendum on Estimands and Sensitivity Analysis in Clinical Trials. 2019.
2. Kane MT. Validating the Interpretations and Uses of Test Scores. Journal of Educational Measurement. 2013;50(1):1–73.
3. Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: a practical guide to Kane's framework. Medical Education. 2015;49(6):560–575.
4. Salaudeen O, Reuel A, Ahmed A, et al. Measurement to Meaning: A Validity-Centered Framework for AI Evaluation. 2025.