What Should an Agent Evaluation Report So Someone Else Can Use It?

A table can say:

Agent A — 74%

and still leave most of the evaluation unrecoverable. Was this the base model or a particular scaffold? Which task version? Which tools and permissions? How were crashes and retries handled? What did the grader inspect? How many runs were performed? Which target workflow is the number being used to discuss? These details are often spread across a paper, repository, appendix, benchmark implementation, and private evaluation script. A reusable evaluation record should bring the decision-relevant parts together.

Identify the evaluated system

For an agent, the model name alone is usually incomplete. Record the model version together with the scaffold, prompts or policies that materially define behavior, available tools, permissions, environment, and date. If a tool, model, or scaffold changes, the identity of the evaluated system may have changed as well. This is the system side of the record. The evaluation side should be identified separately: benchmark or task-set version, environment, grader, tests, parser, run budget, retry policy, and other conditions that determine how behavior becomes a result.

State the task frame and run policy

A sample only supports a population-level claim when the target population and the frame from which the evaluated tasks were obtained are clear. Record:

Target task population: the work the claim is intended to describe.
Evaluation task frame: the tasks that could actually enter the evaluation.
Selection process: how the evaluated tasks were sampled or selected.

The run policy also belongs in the record. Agent runs are interrupted by timeouts, budget limits, tool failures, harness crashes, human interventions, unanswered clarification requests, and partial completion. Clinical trial methodology treats analogous intercurrent events as part of the estimand because different handling rules change the reported result [1]. For agent evaluation, report in advance how each material event is counted, retried, excluded, or otherwise handled.

Preserve the evidence behind the score

The report should make it possible to inspect how observed work became the reported number. Useful fields include the number of tasks and runs, the artifacts or trajectories retained, the grader and reference standard, uncertainty estimates, repeated-run behavior, sensitivity to prompts or other evaluation choices, and material failure categories. A final-answer score can be useful, but decision-facing evaluation often requires the work product or trajectory that explains the score. This is especially important when protocol exploitation, grader error, or downstream reviewability is plausible. The evidence record does not require every artifact to be public. Private evaluations can still state what was retained, who reviewed it, and what evidence supports the reported conclusion.

Connect the result to the claim being made

A result should be paired with the claim it is being used to support. The record should therefore name:

Claim: what is being asserted.
Intended use or decision: what action the claim is meant to influence.
Target workflow: the setting to which the result is being applied.
Demonstrated correspondences: which parts of the evaluation match the target setting.
Material assumptions: which connections remain untested.

Argument-based validity separates scoring, generalization, extrapolation, and decision evidence [2,3]. The record can summarize support at each transition without converting those judgments into one overall validity score.

Report where the evidence stops

A compact evidence review should end with two fields:

Claim ceiling: the strongest conclusion currently supported by the evidence.
Missing evidence / work order: the specific evaluation, comparison, or validation needed to support the next inference.

For example:

Claim under review: "A is ready for workflow X"

Scoring support                    established
Evaluation-domain generalization   established with limitations
Target-setting extrapolation       not established
Decision threshold                 not specified

Claim ceiling:
Performance in the defined evaluation domain
under the evaluated configuration.

Work order:
Compare task frame and tool permissions with workflow X;
validate the grader on known-fault artifacts;
specify the deployment threshold and decision rule.

The work order specifies which evaluation should be conducted next.

Evaluation Evidence Record

The evidence attached to the result allows another person to interpret, review, and update the conclusion.

References

1. International Council for Harmonisation. ICH E9(R1): Addendum on Estimands and Sensitivity Analysis in Clinical Trials. 2019.
2. Kane MT. Validating the Interpretations and Uses of Test Scores. Journal of Educational Measurement. 2013;50(1):1–73.
3. Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: a practical guide to Kane's framework. Medical Education. 2015;49(6):560–575.
4. Bossuyt PM, Reitsma JB, Bruns DE, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527.
5. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336:924–926.
6. Hua Y, Na H, Ayubcha C, Lian L. Design and Report Benchmarks for Knowledge Work. arXiv:2605.23262. 2026.
7. Hua Y, Na H, Zhou Y, Kalose A, Ayubcha C, Lian L. StagedWorkspace: A Versioned Workspace for Knowledge-Work Agents. arXiv:2608.18050. 2026.