In a move to advance the field of artificial intelligence evaluation, renowned AI researcher François Chollet has announced the formation of the ARC Prize Foundation, a nonprofit organization dedicated to developing benchmarks for testing human-level AI capabilities.
The foundation, co-led by former Salesforce engineering director Greg Kamradt as president, aims to build upon Chollet's existing ARC-AGI (Abstract and Reasoning Corpus for Artificial General Intelligence) benchmark system.
ARC-AGI, introduced in 2019, presents AI systems with puzzle-like challenges requiring them to generate correct pattern solutions using colored squares. The test specifically evaluates an AI's ability to adapt to novel problems outside its training data.
While current AI systems excel at specialized tasks like solving complex mathematical problems, they have struggled with ARC-AGI's broader cognitive challenges. Even the most advanced systems have only managed to solve about one-third of the benchmark's tasks.
The foundation plans to release an enhanced second-generation ARC-AGI benchmark in early 2024, alongside a new competition. Early indicators suggest this updated version will present even greater challenges for current AI systems, including OpenAI's unreleased o3 model.
"You'll know artificial general intelligence is here when the exercise of creating tasks that are easy for regular humans but hard for AI becomes simply impossible," states Chollet, highlighting the foundation's practical approach to measuring AI progress.
The organization has announced plans to build academic networks and establish partnerships with leading AI laboratories to collaborate on industry-wide AGI benchmarks. OpenAI CEO Sam Altman has expressed interest in partnering with the ARC-AGI team for future benchmark development.
As debates continue about the definition and current state of artificial general intelligence, the ARC Prize Foundation positions itself as a key player in establishing clear metrics for evaluating AI advancement toward human-level capabilities.
The foundation will begin its fundraising efforts in January 2024, marking a new chapter in the systematic evaluation of artificial intelligence systems.