Authors: Ayrton San Joaquin, Rokas Gipiškis, and Ze Shen Chin
Update: this working draft has been developed further and published as “Scorecard of AI Benchmark Quality” at the EvalEval Workshop, ACL 2026. That version focuses on benchmarks specifically. We recommend starting there.
This working draft white paper describes how to score an evaluation based on certain qualities, and which indicates the setting the evaluation can be used for.
Effective AI risk assessment relies on the quality of evaluations. Currently, there are large quality differences (construct validity, annotation, etc.) between existing evaluations. We propose a Quality Scorecard for AI Evaluations, a standards-based framework designed to make this diversity easier to navigate by developing quality scores for evaluations and to identify appropriate evaluations for specific safety evaluation contexts, ranging from research to post-deployment. The scorecard employs two main components: dimensions, which provide granular scores of an evaluation under that dimension, and classifications, which correspond to concrete use-cases. By establishing a common language and objective methods, this framework aims to aid in transparency and raise the baseline quality of evaluations across the ecosystem.
This is a working draft, and we expect many details to be changed. We welcome feedback about the scorecard. Please send it to ayrton@aistandardslab.org.