Evaluation Science Brief
How Close Is an Agent Evaluation to the Real Workflow?
A company reports that its agent performs well on a benchmark. The benchmark may be carefully designed, the grader may be accurate, and the result may reproduce across runs. There is still a separate question:
How much does the evaluation setting resemble the setting named in the claim?
If the claim is only:
Agent A achieves 78% on Benchmark B.
the benchmark itself may be enough. If the claim becomes:
Agent A is ready for enterprise research workflows.
the evidence now has to connect Benchmark B to enterprise research workflows. That connection should be evaluated directly.
The benchmark-to-workflow gap
Clinical research distinguishes between efficacy, performance under controlled study conditions, and effectiveness, performance under routine practice conditions. The distinction matters because the study environment can differ from routine use in ways that change the result. PRECIS-2 was developed to describe where a clinical trial sits between explanatory and pragmatic conditions. It examines nine features of the study, including who enters it, where it runs, how the intervention is delivered, how outcomes are measured, and how the analysis handles what happens during the study. For agent evaluations, the corresponding features are:
| Evaluation featureQuestion for an agent evaluation | |
|---|---|
| Tasks | Where did the tasks come from, and how closely do they match incoming work in the target workflow? |
| Task selection | Were tasks selected because they were representative, or because they were available and easy to evaluate? |
| Environment | How similar is the evaluation environment to the production environment? |
| Tools and support | Are tools, permissions, integrations, human assistance, and information access comparable? |
| Agent instructions | Is the evaluation scaffold more prescriptive than the way the agent will actually be used? |
| Interaction rules | Are retries, clarifying questions, escalation, and human intervention handled as they would be in practice? |
| Follow-through | Is success measured at submission, or after the downstream consequences of the work are visible? |
| Outcome | Does the benchmark measure the outcome the operator actually cares about, or a proxy for it? |
| Analysis | Are crashes, timeouts, malformed outputs, and incomplete runs included in the reported result? |
These questions do not determine whether a benchmark is good. They determine what additional evidence is needed before its result is used for a broader claim.
Example: a document-analysis workflow
Suppose an agent is evaluated on 300 document-analysis tasks. It has access to a fixed document corpus and receives a fully specified question. Success is a correct final answer. The agent achieves 82%. The intended deployment is an internal research workflow in which the agent must identify relevant sources, determine what information is missing, use company tools, ask staff for clarification when necessary, preserve sources and calculations, and leave a reviewable work product. The 82% result may provide good evidence about answering questions from a fixed corpus. The benchmark-to-workflow comparison would still identify several untested correspondences:
Tasks: deployment tasks begin before the information need has been fully specified.
Environment: production uses internal systems and permissions absent from the benchmark.
Interaction: the real workflow permits clarification and escalation.
Outcome: the benchmark checks the final answer; the workflow requires a reviewable analysis package.
Follow-through: the benchmark ends at submission; deployment success depends on whether another person can review and use the work.
The 82% result therefore supports a narrower claim than “ready for deployment.” A follow-up evaluation could sample real incoming work, reproduce the production tool environment, preserve incomplete and failed runs, and evaluate the downstream work product.
Three comparisons for workflow correspondence
1. Task availability and representativeness
Many benchmarks are assembled from tasks that can be obtained, specified, and graded reliably. That is reasonable for benchmark construction, but availability does not establish representativeness. An evaluation report should therefore state both:
Target task population: the work the claim refers to.
Evaluation task frame: the tasks that could actually enter the evaluation.
The difference between them is part of the evidence.
2. A benchmark outcome may be a surrogate
An agent benchmark usually measures something because it is observable: a test passes, an answer matches, a rubric score improves. The deployment decision usually concerns something else: useful work completed, errors avoided, analyst time saved, safe actions taken, or decisions improved. Clinical research treats a surrogate endpoint as requiring validation before it substitutes for the outcome of interest [2]. Agent evaluations should at least state the same distinction explicitly:
Measured outcome: what the benchmark directly checks.
Target outcome: what the deployment claim ultimately concerns.
Evidence connecting them: what shows that improvement in the measured outcome predicts improvement in the target outcome.
If the last field is empty, the relationship is an assumption.
3. Human review is part of the target setting
Many deployment arguments assume that a human reviewer will catch the remaining errors. That assumption changes the system being evaluated. A benchmark that measures an autonomous agent is not evidence about an agent-plus-reviewer workflow unless the review process has also been evaluated. The reviewer’s access to evidence, workload, error rate, and ability to recognize agent failures all matter. If human review is part of the deployment claim, it belongs in the evaluation.
A benchmark-to-workflow profile
For a specific deployment claim, record each dimension as:
Matched: the evaluation closely reproduces the relevant workflow condition.
Partially matched: the difference is known and there is some evidence about its effect.
Unmatched: the workflow condition is absent or materially different.
Unknown: the relationship has not been evaluated.
For example:
| Dimension | Status |
|---|---|
| Tasks | Partially matched |
| Environment | Unmatched |
| Tools and permissions | Unmatched |
| Agent instructions | Matched |
| Interaction rules | Unknown |
| Follow-through | Unmatched |
| Outcome | Partially matched |
| Analysis of failures | Matched |
This profile does not need to be collapsed into one number. It records correspondence by dimension and identifies unmatched or unknown conditions requiring further evaluation.
What to report
Before using a benchmark result to support a deployment-facing claim, record:
Claim: what is being asserted.
Target workflow: where the agent is expected to operate.
Evaluation setting: what conditions actually produced the reported result.
Matched conditions: which parts of the workflow are represented directly.
Material differences: which differences could plausibly change performance.
Target outcome: what matters in the real workflow.
Measured outcome: what the evaluation actually checked.
Additional validation: what would test the remaining assumptions.
An evaluation can support a bounded claim even when several fields remain unmatched, and the unmatched fields specify the remaining evaluation work.
References
- Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147. doi:10.1136/bmj.h2147.
- Prentice RL. Surrogate endpoints in clinical trials: definition and operational criteria. Statistics in Medicine. 1989;8(4):431–440. doi:10.1002/sim.4780080407.
- ICH. E9(R1) Addendum on Estimands and Sensitivity Analysis in Clinical Trials to the Guideline on Statistical Principles for Clinical Trials. 2019.
- D'Amour A, Heller K, Moldovan D, et al. Underspecification Presents Challenges for Credibility in Modern Machine Learning. Journal of Machine Learning Research. 2022;23(226):1–61.
- Kapoor S, Stroebl B, Siegel ZS, Nadgir N, Narayanan A. AI Agents That Matter. Transactions on Machine Learning Research. 2025.
The benchmark-to-workflow mapping in this note is an AES working adaptation. It has not been validated as a formal instrument.