Diagnose
Find what current evaluations measure, where they fail, and which gaps limit reliable conclusions.
Fall 2026 · New York City
AI Agent Eval Sci
Symposium on the science of AI agent evaluation.
- Date
- Friday, November 20, 2026
- Location
- New York City · Primarily in person
- Submissions
- Archival papers and non-archival presentations
Evaluation science
Four stages organize the symposium
Build measures
Develop frameworks, constructs, metrics, graders, and validity evidence.
Build systems
Create benchmarks, environments, harnesses, and reproducible evaluation infrastructure.
Apply
Evaluate agents in realistic and consequential settings using real-world evidence.
About the symposium
Advancing how AI agents are evaluated
AI Agent Eval Sci Fall 2026 invites research and practice contributions on the science of evaluating AI agents. We welcome work that improves how agent capabilities, failures, reliability, safety, and real-world performance are measured, as well as evidence from teams building and deploying agent systems.
The meeting connects evaluation methodology with deployment experience across research, industry, and high-stakes practice.
Program
Planned program
The detailed schedule is being finalized. The symposium will include the following components.
Approximately 10 contributed oral presentations
Approximately 20 poster presentations
Invited academic and industry talks
Real-world agent evaluation case studies
Poster and reception session
Structured discussion connecting evaluation methodology with deployment experience
Presentation plans include 12-minute contributed talks with 3 minutes for discussion. Program details remain subject to change.
Speakers
Invited academic and industry perspectives
+
Speakers will be announced
Invited speakers and participating organizations will be added as the program is confirmed.
Call for contributed presentations
Share research, evidence, tools, and real-world evaluation lessons
Authors may submit through either an archival research paper track or a non-archival presentation track. Archival status and presentation format are independent.
Track 01
Archival Research Paper
For original research authors wish to publish in the symposium proceedings.
- 4–6 pages total, including references
- Original work not already published in another archival venue
- Submitted through EasyChair
- Formal peer review, normally with three independent reviews
- Camera-ready paper required for accepted archival work
A proceedings volume in the EPiC Series in Computing is being pursued. Publication is planned and remains subject to formal EasyChair/EPiC approval.
Track 02
Non-Archival Presentation
For work authors wish to present and discuss without publication in archival proceedings.
- 1-page abstract, including references if used
- Ongoing, preliminary, or recently published work is welcome
- Tools, datasets, benchmarks, demonstrations, production failures, and industry cases are eligible
- Previously published work must be clearly identified
- The abstract is used for selection and is not published as an archival paper
This track supports later archival publication eligibility and applied teams presenting real-world evaluation evidence.
Oral presentations
Approximately 10 planned
Selected from both tracks. Priority is given to important evaluation questions, strong evidence, lessons beyond one model or application, important failures or real-world challenges, and distinctive perspectives.
Industry and real-world case studies are fully eligible. A new benchmark or algorithm is not required.
Poster presentations
Approximately 20 planned
Posters may present complete studies, work in progress, tools, datasets, infrastructure, audits, replications, negative findings, real-world failures, published work, or early ideas.
Selection is based on relevance and discussion value, not lower scientific importance.
Topics across the four stages
1. Diagnose the field
Benchmark and leaderboard audits, systematic reviews, replication studies, negative results, contamination, grader instability, harness effects, coverage gaps, reproducibility failures, and mismatches between scores and real-world behavior.
2. Build frameworks and measures
Evaluation frameworks, construct and capability definitions, metrics, rubrics, human evaluation, automated graders, executable verifiers, trajectory-level measures, work-product assessment, uncertainty, reliability, validity, efficiency, cost, and safety measures.
3. Build benchmarks and infrastructure
Benchmarks, realistic environments, evaluation harnesses, tool and state management, trace collection, repeated-run evaluation, versioning, observability, benchmark integrity, scalable grading, reproducibility infrastructure, and action or trajectory audits.
4. Apply to real-world systems
Coding, research, scientific, healthcare, enterprise, workplace, web, computer-use, multimodal, and other tool-using or autonomous agents. Concrete failure cases, production traces, deployment evaluations, and evidence linking formal evaluation to practical failure are encouraged.
Review criteria
Archival papers
- Relevance to agent evaluation science
- Clarity of the evaluation target and claims
- Methodological soundness
- Appropriateness of measurement and experimental design
- Reliability and validity of the evidence
- Reproducibility and reporting quality
- Originality and scientific contribution
- Whether conclusions are supported by the evaluation
Non-archival presentations
- Relevance to the symposium
- Technical or practical substance
- Clarity of the evaluation problem and evidence
- Value to scientific discussion
- Usefulness of lessons for other researchers or practitioners
Industry submissions are assessed on concrete failure evidence, production lessons, evaluation methodology, and generalizable insight. Academic novelty alone is not required.
Presentation and publication policy
- Archival and non-archival submissions may both be selected for oral or poster presentation.
- Authors may select Oral preferred, Poster preferred, or Either. Presentation format is determined separately from archival acceptance.
- Previously published work should be submitted only through the non-archival track.
- Archival papers must be original and satisfy the final proceedings venue requirements.
- At least one author of every accepted contribution must register and present.
- The symposium is primarily in person in New York City. Limited remote presentation may be approved when necessary.
Submission portal
EasyChair link coming soon
The submission link will be posted when the symposium installation is open.
Important dates
Submission timeline
All deadlines are 11:59 PM Anywhere on Earth (AoE).
Call opensAugust 2026
Submission deadlineOctober 2, 2026
NotificationOctober 19, 2026
Camera-ready archival papersNovember 6, 2026
SymposiumNovember 20, 2026
Venue
New York City
The symposium is designed primarily as an in-person meeting. Venue and travel details will be announced after arrangements are finalized.
Limited remote presentation may be approved when necessary.