Follow us Research & Commons
Research Agenda Updated Fall 2026
Agent evaluations take a lot of time and compute, and the numbers they produce are often less solid than they look. For example, on SWE-bench Verified, no entry in the top thirty can be told apart from its neighbor (Liu et al., 2026); in a randomized study, models alone named the relevant medical condition in 95% of cases, but people using the same models did so in at most 35% (Bean et al., 2026). EvalScience aims to map the open questions behind results like these and work on them with researchers and practitioners across fields to make evaluations more reliable and valid, cheaper to run, and safer to base decisions on. The agenda below is a map of the questions we prioritize as of Fall 2026. We update it as work is completed and new challenges emerge.Components of the target evaluation
01PopulationWhich tasks, users and settings
Representative questions
- Would two experts split the same job into the same tasks?
- Which task features change the difference between two agents?
- Do benchmark results predict how agents do on real organizations' work?
02Systems comparedWhat exactly is being evaluated
Representative questions
- Is a retrieval index, a human approver or a persistent memory part of the agent?
- Is there a "model ability" for agentic work apart from the harness it runs in?
- How large are model-harness interactions across many models and harnesses?
- How often does a closed model change behind the same name?
03Assignment and repetitionPairing, randomizing and running again
Representative questions
- Are runs close in time more alike than runs far apart?
- How much run-to-run variation comes from the model, the inference stack, tool timing or the environment?
- How many runs per task does a comparison need?
- Can more runs go to tasks where agents disagree without biasing the result?
04EnvironmentTasks, checks and the world the agent acts in
Representative questions
- How far does a sandbox differ from the real setting it models?
- How exploitable is a benchmark before agents find out?
- How quickly do benchmark defects start to matter as agents improve?
- When a run fails, was it the environment or the agent?
05OutcomeWhat counts as success, and who decides
Representative questions
- Do LLM graders make different errors on different agents?
- How quickly does a grader's validation expire as agents learn its blind spots?
- Do benchmark verdicts predict the outcomes they stand in for?
- Do benchmarks measure what their names say, such as autonomy or reliability?
- Can a stated population, estimand and set of assumptions be turned into tests automatically, and what do the generated tests miss?
06Follow-up and the estimandTime limits and runs without a verdict
Representative questions
- When a run crashes or times out, is the agent or the platform responsible?
- How much do rankings change with the time limit?
- What should a half-finished task count as at the cutoff?
- Who should specify the estimand for each kind of agentic scenario, as clinical areas do through estimand working groups?
07Analysis and reportingFrom runs to a stated conclusion
Representative questions
- How much do submitters' unreported choices of runs and configurations bias leaderboards?
- How can leaderboard results stay valid as new entries keep arriving?
- What do intervals mean when tasks were chosen by hand rather than sampled?
- Which reporting items actually change readers' conclusions?
Claims about users, risk and deployment
08UsersClaims about people working with agents
Representative questions
- How do people's skills and reliance change over months of agent use?
- How do we measure whether a person supervising an agent notices what it did?
- When is a simulated user good enough to replace a real one?
- Who do agents help, and who do they hurt, across levels of expertise?
09Safety, control and securityClaims about what agents should not do
Representative questions
- How can harms rarer than one in ten thousand runs be estimated?
- Do raters agree on harm severity across settings?
- How can red-team results be compared when attackers differ so much in skill?
- Can test results be trusted if a model can tell it is being tested?
- When a monitor flags an agent's action, how often is the flag right, and how often does the agent get around the monitor on the next try?
- How do we measure whether a human stays able to stop or correct an agent once the agent runs for hours rather than minutes?
- Do control protocols that hold up against one model family hold up against the next?
- What is the smallest set of tests that tells a lab whether a new model is safe to put into a given agent scaffold?
10After deploymentClaims about agents in use
Representative questions
- Can deployment logs be trusted when people choose which tasks to give the agent?
- How close do estimates from logs come to a randomized rollout?
- How quickly does pre-release evidence expire as a deployed agent is updated?
11Training environments and reward signalsClaims about what an agent learned from
Agents are now trained in environments with automatic reward, so every weakness of a benchmark grader also becomes a weakness of the training signal.
Representative questions
- How often does the reward in an RL environment pass behavior a domain expert would reject?
- When a verifier only checks part of a task, what does the agent learn to skip?
- Do gains from training in one environment transfer to the real work the environment stands for?
- How can two environments that claim to teach the same skill be compared?
- How do we detect that an agent has learned the environment's quirks instead of the task?
- What should be reported about an environment, its tasks, its verifier, and its coverage, before anyone trains on it?
Where it fails, and how evidence adds up
12Where the target evaluation fails
Representative questions
- How far does behavior differ between marked tests, unmarked tests and real use?
- Can tests be built that an agent cannot tell apart from real use?
- How quickly can behavioral drift be detected?
- How do we separate gains from learning the evaluation tasks from gains that transfer?
- How much does an agent's score change with the share and type of other agents present?
13From single evaluations to knowledge
Representative questions
- How often do published agent comparisons hold when rerun on fresh samples?
- Do rankings agree across benchmarks that claim to measure the same thing?
- How different are developer-reported and independent results for the same systems?
We update this agenda each semester. Suggest a direction below.
Suggest a directionInterested in working on one of these questions, as a researcher, a mentor or a funder?
Get involved