This is an MLCommons AI Risk and Reliability Working Group publication. AI Standards Lab contributed through the participation of Rokas Gipiškis and Ze Shen Chin as co-authors.
This report introduces the Agentic Product Maturity Ladder V0.1, a systematic benchmarking framework for assessing AI agent reliability across specific tasks. The ladder addresses a critical deployment challenge: the absence of standardised, trustworthy benchmarks that inform real-world decision-making about agentic systems. The framework defines six progressive maturity levels (R0–R5), each testing specific principles from basic capability through to security and reliability.
The research team examined current agent performance across four task domains: airline booking, retail order management, mobile telephone tech support, and vending machine management. Initial results show that whilst research benchmarks exist, no agents have yet achieved performance thresholds warranting transition to R1 (Capable) product benchmarking. The framework introduces a resource-efficient approach by conditionally developing sophisticated industry-standard benchmarks only when research results indicate market readiness, preventing wasteful testing of advanced principles when basic requirements remain unmet.
This work provides deployers with structured criteria for agent adoption decisions, helps developers understand reliability requirements across maturity stages, and offers the AI governance community a foundation for standardised agent evaluation. The benchmarking programme will expand throughout 2026 as agentic capabilities mature and additional task domains are assessed.
Paper authors: Sean McGregor, Deepak Nathani, Lama Saouma, Fazl Barez, Armstrong Foundjem, Tuesday, Aakash Gupta, Jake Thomas, Vassil Tashev, Tianhao Li, Victor Lu, Faiza Khan Khattak, Medha Bankhwal, Murali Emani, Jacqueline Stetson, James Ezick, Jason Stanley, Joachim Baumann, Rokas Gipiškis, Ravishankar K. Iyer, Roman Eng, Kihyuk Nam, William Bartholomew, Alexandre Drouin, Benjamin Larsen, Chin Ze Shen, Daniel Herde, Arihant Chadda, Malek Ben Salem, Daniel Kang, Kurt Bollacker, Mark Watson, and Peter Mattson