Tag: Research Papers
-
Principles and Guidelines for Randomized Controlled Trials in AI Evaluation
We have published a paper setting out five principles and 33 guidelines for randomised controlled trials measuring how AI access affects human behaviour, performance, or wellbeing, filling a gap in shared methodological standards.
-
The Case for ESM3 as a General-Purpose AI Model with Systemic Risk Under the EU AI Act
We examine whether biological foundation models like ESM3 meet the EU AI Act’s criteria for systemic risk. Despite exceeding relevant compute thresholds, ESM3 falls outside current classification guidance because it does not generate human language. We propose four remedies to close this gap.
-
Scorecard of AI Benchmark Quality
We introduce the “Scorecard of AI Benchmark Quality” to grade benchmarks across a set of dimensions reflecting its quality. We then use those scores to map the benchmark to appropriate use-cases. We presented our paper at the Evaluating Evaluations (EvalEval) Workshop of ACL 2026 in San Diego.
-
Open Problems in AI Incident Governance
We examine AI incident governance across regulatory and independent frameworks, identify open problems in how incidents are defined, classified, monitored, and reported, and propose monitoring and reporting principles, guidelines, and a reporting template.
-
Defining AI Models and AI Systems: A Framework to Resolve the Boundary Problem
AI regulation assigns distinct obligations to providers of AI models and AI systems, but the lack of clear, consistent definitions for “AI model” and “AI system” creates ambiguity across the value chain. This paper surveys the definitions used in academic literature and regulatory documents, and proposes conceptual and operational definitions for drawing a principled boundary…
-
Deprecating Benchmarks: Criteria and Framework
As AI models rapidly advance, many benchmarks become outdated or flawed yet continue to be used, inflating performance claims and obscuring safety concerns. This paper introduces criteria and a framework for deprecating inadequate benchmarks, with recommendations for developers, policymakers, and governance actors on how to maintain rigorous evaluation standards.
-
Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems
A comprehensive, public domain catalog of AI risks and safety measures designed to support global AI regulation and standards development. This resource documents risk sources and management measures across the entire AI lifecycle, from development through deployment.