Evaluation Science Brief
Is This Agent Result Reproducible?
Suppose the same agent scores 74% today, 69% tomorrow, 81% under another prompt, and 62% when another team implements the evaluation. Which result should another evaluator expect to reproduce? That depends on which conditions are supposed to stay fixed, and running one setup five times answers only the first of these questions.
Why rerunning is not enough
Analytical measurement separates repeatability (same setup, short interval), intermediate precision (conditions that change within one organization, such as the day or the machine), and reproducibility (an independent lab or implementation) [1]. Repeated runs only provide evidence for the first. For agents, scores also move with the task sample, the prompt wording [2], the scaffold, and the model version behind an API.
Step 1: Decide what the result should survive
Write down the conditions the claim is meant to cover. A claim about "Agent A on this benchmark" should survive reruns and a fresh task sample. A claim about "Agent A for code review" should also survive reasonable changes in prompt and scaffold. Conditions that define the system, such as the model, are not varied.
Step 2: Vary those conditions on purpose
Change the important conditions in a planned way instead of repeating one setup. Two prompts, two tool versions, and retry on or off give eight combinations, which say more about stability than eight repeats of a single setup. Also draw a second task sample from the same source, because repeating the same 100 tasks only shows variation on those tasks.
Step 3: Report the range across reasonable setups
When several setups are equally defensible, report the result across all of them, as multiverse and specification-curve analyses do for analytic choices [3,4]. Choose the setups before seeing the results, so that a favorable one is not picked afterwards.
Step 4: Recheck after updates
Models, tools, and graders change. Keep a small reference subset, rerun it after each update, and compare the result with a tolerance set in advance.
Report which conditions the result survived
For example:
Performance ranged from 63% to 78% across five pre-specified setups. Repeatability established; a second task sample changed the score by 2 points; independent implementation not yet tested.
| Level | What was varied |
|---|---|
| Repeatability | Repeated runs of one setup |
| Task sample | A second sample from the same task source |
| Configuration | Prompts, scaffold, tools, retry policy |
| Reproducibility | Independent implementation |
| Version stability | Rerun after model, tool, or grader updates |
References
- International Council for Harmonisation. ICH Q2(R2): Validation of Analytical Procedures. 2023.
- Mizrahi M, Kaplan G, Malkin D, et al. State of What Art? A Call for Multi-Prompt LLM Evaluation. Transactions of the Association for Computational Linguistics. 2024;12:933–949.
- Steegen S, Tuerlinckx F, Gelman A, Vanpaemel W. Increasing Transparency Through a Multiverse Analysis. Perspectives on Psychological Science. 2016;11(5):702–712.
- Simonsohn U, Simmons JP, Nelson LD. Specification curve analysis. Nature Human Behaviour. 2020;4:1208–1214.