Evaluating Intelligence via Trial and Error
Intelligence is a crucial trait for species to find solutions within a
limited number of trial-and-error attempts. Building on this idea, we introduce
Survival Game as a framework to evaluate intelligence based on the number of
failed attempts in a trial-and-error process. Fewer failures indicate higher
intelligence. When the expectation and variance of failure counts are both
finite, it signals the ability to consistently find solutions to new
challenges, which we define as the Autonomous Level of intelligence. Using
Survival Game, we comprehensively evaluate existing AI systems. Our results
show that while AI systems achieve the Autonomous Level in simple tasks, they
are still far from it in more complex tasks, such as vision, search,
recommendation, and language. While scaling current AI technologies might help,
this would come at an astronomical cost. Projections suggest that achieving the
Autonomous Level for general tasks would require 10^{26} parameters. To put
this into perspective, loading such a massive model requires so many H100 GPUs
that their total value is 10^{7} times that of Apple Inc.'s market value.
Even with Moore's Law, supporting such a parameter scale would take 70 years.
This staggering cost highlights the complexity of human tasks and the
inadequacies of current AI technologies. To further investigate this
phenomenon, we conduct a theoretical analysis of Survival Game and its
experimental results. Our findings suggest that human tasks possess a
criticality property. As a result, Autonomous Level requires a deep
understanding of the task's underlying mechanisms. Current AI systems, however,
do not fully grasp these mechanisms and instead rely on superficial mimicry,
making it difficult for them to reach an autonomous level. We believe Survival
Game can not only guide the future development of AI but also offer profound
insights into human intelligence.
67d1080b2264403cbf36b0ad
Jingtao Zhan
claimed_verified
2025-03-12T10:45:08.742Z
62f8b1e7c4817cfc07532fbf
/avatars/b21661f868fd968c0ad0910da2a04b65.svg
Jingtao Zhan
user
jingtao
67d1080b2264403cbf36b0ae
Jiahao Zhao
67d1080b2264403cbf36b0af
Jiayu Li
67d1080b2264403cbf36b0b0
Yiqun Liu
67d1080b2264403cbf36b0b1
Bo Zhang
67d1080b2264403cbf36b0b2
Qingyao Ai
67d1080b2264403cbf36b0b3
Jiaxin Mao
67d1080b2264403cbf36b0b4
Hongning Wang
67d1080b2264403cbf36b0b5
Min Zhang
67d1080b2264403cbf36b0b6
Shaoping Ma
Evaluating Intelligence via Trial and Error
Intelligence is a crucial trait for species to find solutions within a
limited number of trial-and-error attempts. Building on this idea, we introduce
Survival Game as a framework to evaluate intelligence based on the number of
failed attempts in a trial-and-error process. Fewer failures indicate higher
intelligence. When the expectation and variance of failure counts are both
finite, it signals the ability to consistently find solutions to new
challenges, which we define as the Autonomous Level of intelligence. Using
Survival Game, we comprehensively evaluate existing AI systems. Our results
show that while AI systems achieve the Autonomous Level in simple tasks, they
are still far from it in more complex tasks, such as vision, search,
recommendation, and language. While scaling current AI technologies might help,
this would come at an astronomical cost. Projections suggest that achieving the
Autonomous Level for general tasks would require 10^{26} parameters. To put
this into perspective, loading such a massive model requires so many H100 GPUs
that their total value is 10^{7} times that of Apple Inc.'s market value.
Even with Moore's Law, supporting such a parameter scale would take 70 years.
This staggering cost highlights the complexity of human tasks and the
inadequacies of current AI technologies. To further investigate this
phenomenon, we conduct a theoretical analysis of Survival Game and its
experimental results. Our findings suggest that human tasks possess a
criticality property. As a result, Autonomous Level requires a deep
understanding of the task's underlying mechanisms. Current AI systems, however,
do not fully grasp these mechanisms and instead rely on superficial mimicry,
making it difficult for them to reach an autonomous level. We believe Survival
Game can not only guide the future development of AI but also offer profound
insights into human intelligence.
Survival Game evaluates AI intelligence via failure counts in trial-and-error processes, highlighting the challenge of achieving Autonomous Level in complex tasks and the critical need for deep task understanding.
Autonomous Level Survival Game criticality property