AI just got a significant boost with the announcement of the ARC Prize Foundation. Spearheaded by François Chollet, a former Google engineer and a luminary in the AI research space, this nonprofit is set to create new AGI benchmarks.
AGI (Artificial General Intelligence), often dubbed the “holy grail” of AI, refers to systems that can perform most tasks humans are capable of. It’s an ambitious target, and Chollet’s new foundation is setting its sights on creating benchmarks that probe AI for truly “human-level” intelligence.
The ARC Prize Foundation isn’t starting from scratch. It will build upon Chollet’s earlier work, particularly the ARC-AGI (Abstract and Reasoning Corpus for Artificial General Intelligence), a test designed to assess an AI’s ability to solve problems outside its training data. The foundation’s goal? To close the gap between AI and human cognitive capabilities, one benchmark at a time.
A Team with Vision and Experience
At the helm of the ARC Prize Foundation is Greg Kamradt, a former engineering director at Salesforce and founder of the AI product studio Leverage. Kamradt will serve as president and board member, bringing both technical expertise and entrepreneurial experience to the table.
Fundraising efforts for the nonprofit are set to kick off later in January. Chollet, in a statement on the foundation’s website, emphasized its broader mission: “We’re growing into a proper nonprofit foundation to act as a useful north star toward artificial general intelligence.”
ARC-AGI: A Benchmark Unlike Any Other
Launched in 2019, ARC-AGI is not your typical AI test. Unlike exams where AI systems like ChatGPT or GPT-4 excel, such as math or coding challenges, ARC-AGI presents unique puzzles that force AI to adapt. These puzzles require the system to generate the correct “answer” grid based on a collection of colored squares, mimicking human problem-solving.
The results? Eye-opening. Even the most advanced AI systems struggled to solve more than a third of the tasks, while a human with no prior training could score over 95%.
Chollet is quick to acknowledge ARC-AGI’s limitations. Some models have managed to brute-force their way to high scores, highlighting flaws in the current version. But these challenges are part of what drives Chollet and his team to innovate further.
OpenAI andAI Benchmarks
The ARC Prize Foundation’s efforts have caught the attention of industry heavyweights like OpenAI. CEO Sam Altman has expressed interest in collaborating with Chollet’s team to develop future benchmarks.
Interestingly, OpenAI’s o3 model was the first to achieve a qualifying score on ARC-AGI. However, this success came with caveats—it required immense computational power, and Chollet maintains that o3 does not exhibit true human-level intelligence.
The foundation plans to launch the second-generation ARC-AGI benchmark in Q1, along with a new competition. Work is also underway for a third edition, ensuring that the benchmarks evolve alongside AI advancements.
Challenges and Opportunities in Bridging the Human-AI Gap
Critics have questioned whether ARC-AGI truly measures progress toward AGI, with some arguing that the definition of AGI itself is too ambiguous. Chollet and his team aim to address these concerns by refining their benchmarks and fostering collaboration within the AI research community.
To further its mission, the ARC Prize Foundation is building an academic network and establishing partnerships with frontier AI labs. These efforts will ensure that their benchmarks remain relevant and rigorous as the field advances.
Why Does This Matter for AI
The pursuit of AGI isn’t just about creating smarter machines, it’s about understanding and replicating the essence of human intelligence. Chollet’s work, through the ARC Prize Foundation, represents a critical step in this journey.
By setting higher standards for AGI benchmarks, the foundation is not only extending the value we get from AI but also ensuring that progress is meaningful and measurable.