Reflection-Bench: probing AI intelligence with reflection
The ability to adapt beliefs or behaviors in response to unexpected outcomes,
reflection, is fundamental to intelligent systems' interaction with the world.
From a cognitive science perspective, this serves as a core principle of
intelligence applicable to both human and AI systems. To address the debate on
the intelligence of large language models (LLMs), we propose Reflection-Bench,
a comprehensive benchmark comprising 7 tasks spanning core cognitive functions
crucial for reflection, including perception, memory, belief updating,
decision-making, prediction, counterfactual thinking, and meta-reflection. We
evaluate the performances of 13 prominent LLMs such as OpenAI o1, GPT-4, Claude
3.5 Sonnet, etc. The results indicate that current LLMs still lack satisfactory
reflection ability. We discuss the underlying causes of these results and
suggest potential avenues for future research. In conclusion, Reflection-Bench
offers both evaluation tools and inspiration for developing AI capable of
reliably interacting with the environment. Our data and code are available at
https://github.com/YabYum/ReflectionBench.
671f35fc2750014020ef8e8f
Lingyu Li
claimed_verified
2024-10-28T08:51:28.542Z
65c1d9413601dac1372b0ffe
/avatars/109f2af09a29b16a884b77c930bb7f48.svg
Lingyu Li
user
LingyuLi
671f35fc2750014020ef8e90
Yixu Wang
671f35fc2750014020ef8e91
Haiquan Zhao
admin_assigned
2024-10-28T15:45:27.152Z
64fecf8dd30d99e75e0a557c
/avatars/6f6ba57711f7717bcab8e3db05a849cb.svg
Haiquan Zhao
user
haidequanbu
671f35fc2750014020ef8e92
Shuqi Kong
671f35fc2750014020ef8e93
Yan Teng
671f35fc2750014020ef8e94
Chunbo Li
671f35fc2750014020ef8e95
Yingchun Wang
admin_assigned
2024-10-28T15:45:58.257Z
66f108310ea83b350d0d7411
/avatars/93a68a13b016a8ae81a3453840addfc3.svg
Yingchun WANG
user
DaenerysW
Reflection-Bench: probing AI intelligence with reflection
The ability to adapt beliefs or behaviors in response to unexpected outcomes,
reflection, is fundamental to intelligent systems' interaction with the world.
From a cognitive science perspective, this serves as a core principle of
intelligence applicable to both human and AI systems. To address the debate on
the intelligence of large language models (LLMs), we propose Reflection-Bench,
a comprehensive benchmark comprising 7 tasks spanning core cognitive functions
crucial for reflection, including perception, memory, belief updating,
decision-making, prediction, counterfactual thinking, and meta-reflection. We
evaluate the performances of 13 prominent LLMs such as OpenAI o1, GPT-4, Claude
3.5 Sonnet, etc. The results indicate that current LLMs still lack satisfactory
reflection ability. We discuss the underlying causes of these results and
suggest potential avenues for future research. In conclusion, Reflection-Bench
offers both evaluation tools and inspiration for developing AI capable of
reliably interacting with the environment. Our data and code are available at
https://github.com/YabYum/ReflectionBench.
Reflection-Bench is a benchmark for evaluating the reflection abilities of large language models across various cognitive functions, highlighting their current limitations and future research directions.
large language models LLMs Reflection-Bench perception memory belief updating decision-making prediction counterfactual thinking meta-reflection