Evaluation Science Brief
Is This Agent Good Enough for the Decision?
Suppose an evaluation reports:
Agent A: 74% success.
A procurement requirement says:
Minimum acceptable performance: 70%.
Whether the agent passed depends on questions the two numbers do not answer. How uncertain is 74%? What kinds of failures make up the other 26%? Is 70% based on the consequences of error, or is it a convenient cutoff?
Why a score above the threshold is not enough
An evaluation estimates how well an agent performs, while deployment is a separate judgment. Clinical guidelines keep the two apart. They first rate how certain the evidence is, then make a recommendation that also weighs benefits, harms, and resources [1,2]. Uncertainty matters most near the threshold. If the 95% interval around 74% reaches below 70%, the point estimate alone cannot settle whether the requirement is met [3].
Step 1: Set the threshold from the consequences
Ask what a wrong decision costs in each direction. Trusting an agent that makes undetected errors in research summaries costs something different from rejecting one that would have saved review time. Volume matters too. An error rate that is fine at 10 uses a month may be unacceptable at 100,000 a day. Decision-curve analysis in medicine ties thresholds to consequences in the same way [4].
Step 2: Write the decision rule before seeing the result
State in advance how uncertainty will be handled. A common rule is to accept the agent only if the lower end of the 95% interval clears the threshold, which conformity assessment calls guard banding [3]. Under this rule, 74% with an interval of 66% to 82% does not pass a 70% requirement.
Step 3: Look at which failures make up the rest
Two agents with the same mean can call for different decisions. One may fail on a known type of task that can be routed to people. Another may fail rarely but badly. Sort a sample of failures by type and severity before deciding.
Step 4: Decide whether more evaluation is worth it
If the result sits close to the threshold, ask whether more data could change the decision. If it could, run the evaluation that targets that uncertainty. If it could not, more benchmarking adds little.
Report the decision with its inputs
| Field | Record |
|---|---|
| Decision | What action is being considered, and for which workflow |
| Estimate | Point estimate and 95% interval |
| Threshold | Value and how it was derived from consequences |
| Decision rule | How uncertainty is handled at the threshold |
| Failure structure | Types and severity of the remaining failures |
| Comparator | What the current process or alternatives achieve |
References
- Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–926.
- Alonso-Coello P, Schünemann HJ, Moberg J, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016;353:i2016.
- Joint Committee for Guides in Metrology. Evaluation of measurement data: The role of measurement uncertainty in conformity assessment (JCGM 106:2012). 2012.
- Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making. 2006;26(6):565–574.