GAIA: a benchmark for General AI Assistants
We introduce GAIA, a benchmark for General AI Assistants that, if solved,
would represent a milestone in AI research. GAIA proposes real-world questions
that require a set of fundamental abilities such as reasoning, multi-modality
handling, web browsing, and generally tool-use proficiency. GAIA questions are
conceptually simple for humans yet challenging for most advanced AIs: we show
that human respondents obtain 92\% vs. 15\% for GPT-4 equipped with plugins.
This notable performance disparity contrasts with the recent trend of LLMs
outperforming humans on tasks requiring professional skills in e.g. law or
chemistry. GAIA's philosophy departs from the current trend in AI benchmarks
suggesting to target tasks that are ever more difficult for humans. We posit
that the advent of Artificial General Intelligence (AGI) hinges on a system's
capability to exhibit similar robustness as the average human does on such
questions. Using GAIA's methodology, we devise 466 questions and their answer.
We release our questions while retaining answers to 300 of them to power a
leader-board available at https://huggingface.co/gaia-benchmark.
655ec78fa3e81145767e66a4
Grégoire Mialon
admin_assigned
2023-11-23T10:41:30.419Z
6458c976f1f5263c2ef0514c
/avatars/0cab7449293be91a4fa08c23533b1b1a.svg
Grégoire Mialon
user
gregmialz
655ec78fa3e81145767e66a5
Clémentine Fourrier
extracted_confirmed
2024-06-28T19:57:36.463Z
6202a599216215a22221dea9
https://cdn-avatars.huggingface.co/v1/production/uploads/1644340617257-noauth.png
Clémentine Fourrier
user
clefourrier
655ec78fa3e81145767e66a6
Craig Swift
admin_assigned
2023-11-23T10:41:39.740Z
6467fee96192d391422669d5
/avatars/595133486c0d73eb21138cbd0a49ae48.svg
Craig Swift
user
SwiftyOS
655ec78fa3e81145767e66a7
Thomas Wolf
admin_assigned
2023-11-23T10:41:53.000Z
5df7e9e5da6d0311fd3d53f9
https://cdn-avatars.huggingface.co/v1/production/uploads/1583857746553-5df7e9e5da6d0311fd3d53f9.jpeg
Thomas Wolf
user
thomwolf
655ec78fa3e81145767e66a8
Yann LeCun
admin_assigned
2023-11-23T10:52:30.928Z
64ed0b8c2203a126eb1a5b9a
/avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
Yann LeCun
user
ylecun
655ec78fa3e81145767e66a9
Thomas Scialom
admin_assigned
2023-11-23T10:52:47.274Z
60c0e0c06c0791283058f0fb
/avatars/84c0fe390a171e68b79a8274405ebf06.svg
Scialom
user
ThomasNLG
GAIA: a benchmark for General AI Assistants
We introduce GAIA, a benchmark for General AI Assistants that, if solved,
would represent a milestone in AI research. GAIA proposes real-world questions
that require a set of fundamental abilities such as reasoning, multi-modality
handling, web browsing, and generally tool-use proficiency. GAIA questions are
conceptually simple for humans yet challenging for most advanced AIs: we show
that human respondents obtain 92\% vs. 15\% for GPT-4 equipped with plugins.
This notable performance disparity contrasts with the recent trend of LLMs
outperforming humans on tasks requiring professional skills in e.g. law or
chemistry. GAIA's philosophy departs from the current trend in AI benchmarks
suggesting to target tasks that are ever more difficult for humans. We posit
that the advent of Artificial General Intelligence (AGI) hinges on a system's
capability to exhibit similar robustness as the average human does on such
questions. Using GAIA's methodology, we devise 466 questions and their answer.
We release our questions while retaining answers to 300 of them to power a
leader-board available at https://huggingface.co/gaia-benchmark.
GAIA benchmarks general AI assistants using real-world questions that challenge both reasoning and multi-modality handling, showcasing a significant gap between human and AI performance.
multi-modality handling web browsing tool-use proficiency GPT-4 LLMs Artificial General Intelligence (AGI)