OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for
  Superintelligent AI
The evolution of Artificial Intelligence (AI) has been significantly
accelerated by advancements in Large Language Models (LLMs) and Large
Multimodal Models (LMMs), gradually showcasing potential cognitive reasoning
abilities in problem-solving and scientific discovery (i.e., AI4Science) once
exclusive to human intellect. To comprehensively evaluate current models'
performance in cognitive reasoning abilities, we introduce OlympicArena, which
includes 11,163 bilingual problems across both text-only and interleaved
text-image modalities. These challenges encompass a wide range of disciplines
spanning seven fields and 62 international Olympic competitions, rigorously
examined for data leakage. We argue that the challenges in Olympic competition
problems are ideal for evaluating AI's cognitive reasoning due to their
complexity and interdisciplinary nature, which are essential for tackling
complex scientific challenges and facilitating discoveries. Beyond evaluating
performance across various disciplines using answer-only criteria, we conduct
detailed experiments and analyses from multiple perspectives. We delve into the
models' cognitive reasoning abilities, their performance across different
modalities, and their outcomes in process-level evaluations, which are vital
for tasks requiring complex reasoning with lengthy solutions. Our extensive
evaluations reveal that even advanced models like GPT-4o only achieve a 39.97%
overall accuracy, illustrating current AI limitations in complex reasoning and
multimodal integration. Through the OlympicArena, we aim to advance AI towards
superintelligence, equipping it to address more complex challenges in science
and beyond. We also provide a comprehensive set of resources to support AI
research, including a benchmark dataset, an open-source annotation platform, a
detailed evaluation tool, and a leaderboard with automatic submission features.
66724205c12108af6ccd1d3b
Zhen Huang
admin_assigned
2024-06-19T20:49:48.170Z
6416af7a58a690df1040f422
/avatars/224d4015382063e713a99f96bf332722.svg
Zhen Huang
user
Benjamin02
66724205c12108af6ccd1d3c
Zengzhi Wang
admin_assigned
2024-06-19T21:00:36.085Z
62cbeb2d72dfd24b86bdf977
https://cdn-avatars.huggingface.co/v1/production/uploads/62cbeb2d72dfd24b86bdf977/UcGYYSBNrCvPM5K9v-sro.png
Zengzhi Wang
user
SinclairWang
66724205c12108af6ccd1d3d
Shijie Xia
admin_assigned
2024-06-19T21:00:41.740Z
65900d4ff5a209eeac08b463
https://cdn-avatars.huggingface.co/v1/production/uploads/65900d4ff5a209eeac08b463/PJNNBRJIk1qR24oaRLTex.jpeg
shijie xia
user
seven-cat
66724205c12108af6ccd1d3e
Xuefeng Li
admin_assigned
2024-06-19T21:00:59.064Z
64e3562342d8e2c1c69f64b3
/avatars/1ccf22d60deec213bbe069d30811efbe.svg
xuefengli
user
xuefengli
66724205c12108af6ccd1d3f
Haoyang Zou
admin_assigned
2024-06-19T21:01:05.197Z
65011b18af1e4021439af578
/avatars/ae1632c800ceeb43aa223e432f70bc6d.svg
Haoyang Zou
user
alanyoung058
66724205c12108af6ccd1d40
Ruijie Xu
admin_assigned
2024-06-19T21:01:11.785Z
6537cdda25f780fed2375a7b
/avatars/6309fbcbb97cd63957ed7429bbe311d0.svg
Ruijie Xu
user
GardenStrayer
66724205c12108af6ccd1d41
Run-Ze Fan
claimed_verified
2024-06-19T07:38:50.815Z
616bfc2b40e2f69baa1c7add
https://cdn-avatars.huggingface.co/v1/production/uploads/616bfc2b40e2f69baa1c7add/Os7_qgMei-2lRVelrOG7B.jpeg
Run-Ze Fan
user
Vfrz
66724205c12108af6ccd1d42
Lyumanshan Ye
66724205c12108af6ccd1d43
Ethan Chern
admin_assigned
2024-06-19T21:01:22.700Z
64bb5f9d8e051085bace4d1e
/avatars/15ccbb78c6131dfe46b7a9d8e7d1a31f.svg
Ethan Chern
user
ethanchern
66724205c12108af6ccd1d44
Yixin Ye
66724205c12108af6ccd1d45
Yikai Zhang
admin_assigned
2024-06-19T21:01:51.271Z
63e8b792ca4fc7d30de6975b
/avatars/57237f54d61d479df15209497a3f531e.svg
Yikai Zhang
user
Arist12
66724205c12108af6ccd1d46
Yuqing Yang
claimed_verified
2025-05-26T08:50:10.437Z
65c0de12efbb14b39c97f78e
/avatars/18485f79427a35bd9e19f71b67c88dce.svg
Yuqing Yang
user
ayyyq
66724205c12108af6ccd1d47
Ting Wu
66724205c12108af6ccd1d48
Binjie Wang
66724205c12108af6ccd1d49
Shichao Sun
66724205c12108af6ccd1d4a
Yang Xiao
66724205c12108af6ccd1d4b
Yiyuan Li
66724205c12108af6ccd1d4c
Fan Zhou
claimed_verified
2024-06-19T20:20:51.598Z
628f6e5ab90dde28ef57d293
https://cdn-avatars.huggingface.co/v1/production/uploads/628f6e5ab90dde28ef57d293/AxNzR2nvrND6Rf3RPkYMk.jpeg
Fan Zhou
user
koalazf99
66724205c12108af6ccd1d4d
Steffi Chern
admin_assigned
2024-06-19T21:03:14.603Z
64b370fe6d953e7c75ede314
https://cdn-avatars.huggingface.co/v1/production/uploads/64b370fe6d953e7c75ede314/RdP2q3hGXWE4E2zfSv0KU.png
Steffi Chern
user
steffichern
66724205c12108af6ccd1d4e
Yiwei Qin
66724205c12108af6ccd1d4f
Yan Ma
claimed_verified
2025-05-26T08:50:13.002Z
633fc70529b5a95f6e15a6b7
https://cdn-avatars.huggingface.co/v1/production/uploads/633fc70529b5a95f6e15a6b7/Fzh7wWuqU-fBbzdupOUtF.jpeg
Yan Ma
user
ManTle
66724205c12108af6ccd1d50
Jiadi Su
admin_assigned
2024-06-19T21:03:35.224Z
65437c82d3db5d5336af93ff
https://cdn-avatars.huggingface.co/v1/production/uploads/65437c82d3db5d5336af93ff/wdL4jbfcD8TS3NB77LRrr.jpeg
Jiadi Su
user
JoyBoy-Su
66724205c12108af6ccd1d51
Yixiu Liu
66724205c12108af6ccd1d52
Yuxiang Zheng
66724205c12108af6ccd1d53
Shaoting Zhang
66724205c12108af6ccd1d54
Dahua Lin
admin_assigned
2024-06-19T21:03:49.944Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
66724205c12108af6ccd1d55
Yu Qiao
66724205c12108af6ccd1d56
Pengfei Liu
admin_assigned
2024-06-19T21:03:56.925Z
6144a0c4ff1146bbd84d9865
https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
Pengfei Liu
user
Pengfei
OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for
  Superintelligent AI
The evolution of Artificial Intelligence (AI) has been significantly
accelerated by advancements in Large Language Models (LLMs) and Large
Multimodal Models (LMMs), gradually showcasing potential cognitive reasoning
abilities in problem-solving and scientific discovery (i.e., AI4Science) once
exclusive to human intellect. To comprehensively evaluate current models'
performance in cognitive reasoning abilities, we introduce OlympicArena, which
includes 11,163 bilingual problems across both text-only and interleaved
text-image modalities. These challenges encompass a wide range of disciplines
spanning seven fields and 62 international Olympic competitions, rigorously
examined for data leakage. We argue that the challenges in Olympic competition
problems are ideal for evaluating AI's cognitive reasoning due to their
complexity and interdisciplinary nature, which are essential for tackling
complex scientific challenges and facilitating discoveries. Beyond evaluating
performance across various disciplines using answer-only criteria, we conduct
detailed experiments and analyses from multiple perspectives. We delve into the
models' cognitive reasoning abilities, their performance across different
modalities, and their outcomes in process-level evaluations, which are vital
for tasks requiring complex reasoning with lengthy solutions. Our extensive
evaluations reveal that even advanced models like GPT-4o only achieve a 39.97%
overall accuracy, illustrating current AI limitations in complex reasoning and
multimodal integration. Through the OlympicArena, we aim to advance AI towards
superintelligence, equipping it to address more complex challenges in science
and beyond. We also provide a comprehensive set of resources to support AI
research, including a benchmark dataset, an open-source annotation platform, a
detailed evaluation tool, and a leaderboard with automatic submission features.
OlympicArena, a benchmark for evaluating AI's cognitive reasoning across disciplines, demonstrates current models' limitations and provides resources to advance AI towards superintelligence.
Large Language Models LLMs Large Multimodal Models LMMs AI4Science cognitive reasoning OlympicArena bilingual problems text-only interleaved text-image cognitive reasoning abilities complexity interdisciplinary nature data leakage advanced models GPT-4o multimodal integration