Is This Agent Result Reproducible?

Suppose the same agent scores 74% today, 69% tomorrow, 81% under another prompt, and 62% when another team implements the evaluation. Which number is its performance? The answer depends on the claim. A claim about one frozen configuration requires less evidence than a claim that the result is stable across implementations or operational conditions.

Repeatability and reproducibility are different claims

Analytical procedure validation distinguishes several levels of precision [1]. The corresponding levels in agent evaluation are:

Repeatability: same system, same scaffold, same environment, short interval; re-run the evaluation under conditions intended to be effectively identical.
Intermediate precision: same evaluation process, but vary conditions that can change within one organization, such as seed, day, operator, or infrastructure instance.
Reproducibility: independent implementation, laboratory, or evaluation environment.

When an evaluation reports repeated runs of one frozen configuration, it provides repeatability-level evidence. It does not by itself establish reproducibility.

Seeds are one source of variation

Agent evaluations have several plausible variance components:

Running one configuration five times estimates one component of this variation. It tells us little about the others.

Treat evaluation choices as experimental factors

List the evaluation choices that could plausibly change the result, then vary the important ones deliberately. For example:

Prompt:       A / B
Tool version: 1 / 2
Retry policy: off / on

Eight planned combinations can be more informative for a robustness claim than eight repetitions of one setup. The design should identify which factors the claim assumes should be stable and test those factors efficiently. This also helps separate control factors, which define the intended system, from noise factors, which should not materially change the conclusion.

Reproduce the task sample as well as the run

A second team can reproduce the execution protocol and still obtain a different result because it evaluated a different set of tasks. Task sampling and run-to-run stochasticity answer different questions. Repeating the same 100 tasks estimates variation conditional on those tasks. Drawing another 100 tasks asks whether the result generalizes across the task frame. The evaluation report should therefore distinguish the task population, the task frame, and the realized sample. When the benchmark is a fixed finite set, the claim should say so. When the claim concerns a broader class of work, task-sampling variation belongs in the reproducibility analysis. More repeated runs cannot repair a task frame that omits important parts of the target work. For independent reproduction, the implementation boundary should also be explicit. A team that reuses the original harness is checking a different source of variation from a team that reimplements the published specification. Both are useful, but they support different claims.

Report the distribution across defensible specifications

Prompt and harness choices create a second problem. Several configurations may be defensible, and choosing one after seeing the results can make a fragile result look stable. Multiverse analysis and specification-curve analysis were developed to show how conclusions vary across defensible analytic choices [3,4]. The same analysis can be applied to prompts, scaffolds, tool sets, parsers, or retry policies. Instead of reporting only one selected configuration, an evaluation can state:

Performance ranged from 63–78% across five pre-specified defensible configurations.

The range indicates whether the conclusion depends on a specific implementation choice.

Preserve version identity

Agent systems and their dependencies change quickly. A reproducibility package should record the model version, scaffold commit or release, benchmark version, tool and API versions, grader version, date, and run policy. Otherwise, a later team may faithfully follow the written protocol while evaluating a different system. Version stability can also be tested explicitly: rerun a reference subset after a model, tool, grader, or benchmark change and record whether the result remains within a pre-specified tolerance.

Reproducibility Profile

A compact result can therefore read:

Reproducibility status: Repeatability established; configuration robustness partially tested; independent reproducibility unknown.

The status identifies which forms of stability have been tested.

References

1. International Council for Harmonisation. ICH Q2(R2): Validation of Analytical Procedures. 2023.
2. Mizrahi M, Kaplan G, Malkin D, et al. State of What Art? A Call for Multi-Prompt LLM Evaluation. Transactions of the Association for Computational Linguistics. 2024;12:933–949.
3. Steegen S, Tuerlinckx F, Gelman A, Vanpaemel W. Increasing Transparency Through a Multiverse Analysis. Perspectives on Psychological Science. 2016;11(5):702–712. doi:10.1177/1745691616658637.
4. Simonsohn U, Simmons JP, Nelson LD. Specification curve analysis. Nature Human Behaviour. 2020. doi:10.1038/s41562-020-0912-z.
5. Rabanser S, Kapoor S, Kirgis P, et al. Towards a Science of AI Agent Reliability. ICML. 2026.
6. Yao S, Shinn N, Razavi P, Narasimhan K. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. ICLR. 2025.