Diagnose
Identify what current evaluations measure, where they fail, and which gaps limit reliable conclusions.
Fall 2026 · New York City
Agent Evaluation Science
A one-day symposium on the methods, measures, systems, and evidence used to evaluate AI agents.
- Date
- Friday, November 20, 2026
- Location
- Downtown Manhattan · New York City
- Submissions
- Short papers and 2-page extended abstracts via OpenReview
Attendance is free. Paper submission is not required to register.
Explore the symposiumScientific evaluation◆Construct validity◆Measurement reliability◆Benchmark integrity◆Agent safety◆Grader design◆Failure analysis◆Evaluation infrastructure◆Real-world evidence◆Reproducible systems◆
Why this symposium
Scientific methods and evidence for agent evaluation
AI agents act across tools, environments, and extended workflows. Evaluating them requires evidence beyond a single score or benchmark.
The symposium brings together researchers and practitioners working on measurement, benchmark design, evaluation infrastructure, reliability, safety, and real-world performance.
1focused day
2submission formats
4evaluation stages
Evaluation science agenda
Four stages organize the symposium
Measure
Develop frameworks, constructs, metrics, graders, and evidence for reliability and validity.
Operationalize
Build runnable benchmarks, environments, harnesses, and reproducible evaluation infrastructure.
Apply
Evaluate agents in realistic settings using failure evidence and real-world performance.
Symposium program
Invited talks, contributed research, case studies, and discussion
The program covers methodological questions and evidence from applied evaluation.
01
Invited speakers
Academic and industry speakers working on evaluation methodology and deployed agent systems.
View speakers →02Contributed research
Selected short papers and abstracts presented through focused oral and poster sessions.
View the call →03Evaluation in practice
Case studies, benchmarks, tools, failures, and evidence from real-world agent evaluation.
View program →04Agenda discussion
Discussion of diagnosis, measurement, infrastructure, and real-world application.
View agenda →Call for presentations
Submit work on agent evaluation
Submit evaluation studies, audits, negative results, failure analyses, tools, datasets, production lessons, and real-world evaluation cases. Ongoing and previously published work can also be presented through the extended abstract track.
- Evaluation studies
- Audits and replications
- Negative results
- Failure analyses
- Tools and datasets
- Production lessons
- Real-world evaluation cases
Industry and real-world submissions are assessed on documented evidence, production lessons, and findings relevant beyond a single system or setting. A new benchmark or algorithm is not required.
Submissions open
August 252026Abstract registration deadline
October 202026Full submission deadline
October 252026Notification of acceptance
November 52026Symposium
November 202026Affiliations represented
Speakers and committee members are affiliated with universities, research institutes, and industry organizations.
- Agent Evaluation Science
- Harvard University
- Massachusetts General Hospital
- BRIDGE GenAI Lab
- Princeton University
- Cornell University · Cornell Tech
- The University of Texas at Austin
- University of Technology Sydney
- Australian Artificial Intelligence Institute
- University of Alberta
- Amii
- Meta Superintelligence Labs
- Sony AI
- Raycaster
- Google DeepMind
- Massachusetts Institute of Technology
- MIT Media Lab
- Columbia University
- DAPLab
- New York University
- Cohere Labs
- J.P. Morgan
- IBM Research
- Microsoft Research
- Mercor
- University of Illinois Urbana-Champaign
- Dartmouth College
Sponsored by
Affiliations describe speakers’ and committee members’ institutional or organizational connections. They do not indicate sponsorship.
Agent Evaluation Science Fall 2026Friday, November 20 · Downtown Manhattan, New York City
Meet the committee