Evaluation Science Brief
How Close Is an Agent Evaluation to the Real Workflow?
Suppose an agent is evaluated on 300 document-analysis tasks. It receives a fixed document corpus and a fully specified question, success is a correct final answer, and the agent scores 82%. The team wants to use it in an internal research workflow, where the agent has to find its own sources, ask staff when information is missing, and leave an analysis that a colleague can review. Should the team expect something close to 82%?
Why the score can overstate real performance
Benchmarks are built from tasks that are easy to collect and easy to grade. In this example the benchmark does part of the work for the agent: it supplies the right documents, states the question precisely, and accepts a final answer without asking whether anyone could review it. Clinical research knows the same problem as the gap between efficacy, measured under controlled trial conditions, and effectiveness in routine practice, and PRECIS-2 was designed to make the differences between the two explicit [1]. The outcome can differ as well. A correct final answer stands in for a reviewable analysis, and a stand-in outcome needs evidence before it can replace the real one [2].
None of this says how large the gap is. A benchmark can differ from the workflow in many ways and still predict it well, or differ in one way and predict it badly. The only way to know is to measure.
Step 1: Pick the differences most likely to matter
Start by comparing the benchmark with the workflow and writing down where they differ. The table below covers the usual places.
| Feature | Question |
|---|---|
| Tasks | Where did the tasks come from, and how closely do they match incoming work? |
| Information | Does the agent receive what it needs, or does it have to find it? |
| Tools and access | Are the tools and permissions the same as in production? |
| Interaction | Can the agent ask for clarification or hand off to a person? |
| Outcome | Is success the outcome the team cares about, or a stand-in for it? |
| Human review | Will a person check the work in practice, and is that part of the evaluation? |
Then pick the one or two differences most likely to change the score. In the example, these are that the agent does not have to find its own documents and that a final answer is accepted in place of a reviewable analysis.
Step 2: Test the agent on a sample of real tasks
Take a sample of tasks from the work the agent will actually do, for example 30 requests that came in over the past month. Run the agent on them in the real environment, or in shadow mode alongside the people who currently do the work, and have those people grade the results. Suppose the agent succeeds on 18 of 30 (60%, with a 95% interval of 42% to 75%). The interval is wide because the sample is small, but it already shows that 82% is too optimistic for this workflow.
Step 3: Remove the benchmark's advantages one at a time
The real-task sample shows the size of the gap but not its cause. To find the cause, change the benchmark so that it removes one advantage at a time. For example, give the agent the whole document store in place of the relevant files, or grade the analysis on whether a colleague can follow it in place of the final answer. If the score falls from 82% to 70% when the agent has to find its own documents, that difference explains about half of the gap, and the benchmark can be improved by keeping it.
Report the gap alongside the score
The evaluation report should state both numbers and what was learned about the difference:
Benchmark: 82%. Sample of 30 real workflow tasks: 60% (95% interval 42–75%). Requiring the agent to find its own documents lowered the benchmark score to 70%.
A reader can then decide whether the benchmark is good enough for the decision at hand, and the next evaluation knows which difference to fix first.
References
- Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147.
- Prentice RL. Surrogate endpoints in clinical trials: definition and operational criteria. Statistics in Medicine. 1989;8(4):431–440.