KoLA: Carefully Benchmarking World Knowledge of Large Language Models
The unprecedented performance of large language models (LLMs) necessitates
improvements in evaluations. Rather than merely exploring the breadth of LLM
abilities, we believe meticulous and thoughtful designs are essential to
thorough, unbiased, and applicable evaluations. Given the importance of world
knowledge to LLMs, we construct a Knowledge-oriented LLM Assessment benchmark
(KoLA), in which we carefully design three crucial factors: (1) For ability
modeling, we mimic human cognition to form a four-level taxonomy of
knowledge-related abilities, covering 19 tasks. (2) For data, to ensure fair
comparisons, we use both Wikipedia, a corpus prevalently pre-trained by LLMs,
along with continuously collected emerging corpora, aiming to evaluate the
capacity to handle unseen data and evolving knowledge. (3) For evaluation
criteria, we adopt a contrastive system, including overall standard scores for
better numerical comparability across tasks and models and a unique
self-contrast metric for automatically evaluating knowledge hallucination. We
evaluate 21 open-source and commercial LLMs and obtain some intriguing
findings. The KoLA dataset and open-participation leaderboard are publicly
released at https://kola.xlore.cn and will be continuously updated to provide
references for developing LLMs and knowledge-related systems.
648be739152fc5a80d2c7c88
Jifan Yu
admin_assigned
2023-06-19T12:04:33.769Z
632d6e8d2f9c892dcf0ed4bb
/avatars/fba1fb87bcac340a8eee3b9f4fc35bc5.svg
Jifan Yu
user
JovanYu
648be739152fc5a80d2c7c89
Xiaozhi Wang
claimed_verified
2023-06-19T07:15:58.863Z
648c48e7a47a0850f9458b67
/avatars/47d71d80f9901313feb0199c37296389.svg
Xiaozhi Wang
user
wangxz098
648be739152fc5a80d2c7c8a
Shangqing Tu
claimed_verified
2024-12-20T08:40:43.817Z
648c48d8c0ddeee6df5b6d22
/avatars/8706b0b16dfc332b96c91d3ced31bd0b.svg
Shangqing Tu
user
tsq2000
648be739152fc5a80d2c7c8b
Shulin Cao
648be739152fc5a80d2c7c8c
Daniel Zhang-Li
648be739152fc5a80d2c7c8d
Xin Lv
claimed_verified
2024-06-20T07:35:20.032Z
648c64829c935db2b527a764
https://cdn-avatars.huggingface.co/v1/production/uploads/648c64829c935db2b527a764/AnTu1dvpac3hUxLQo8A8K.jpeg
Xin Lv
user
davidlvxin
648be739152fc5a80d2c7c8e
Hao Peng
claimed_verified
2025-03-02T20:20:31.787Z
625a5446f1063e7085d5178a
/avatars/5e78186f13f74b14e01583e06ff6c4dc.svg
Hao Peng
user
Wesleythu
648be739152fc5a80d2c7c8f
Zijun Yao
claimed_verified
2023-10-16T07:25:21.350Z
648c4b46e549be47af1aafcd
https://cdn-avatars.huggingface.co/v1/production/uploads/648c4b46e549be47af1aafcd/YgOHbmUM2EDM-lb7GdqXz.jpeg
Zijun
user
TranSirius
648be739152fc5a80d2c7c90
Xiaohan Zhang
648be739152fc5a80d2c7c91
Hanming Li
648be739152fc5a80d2c7c92
Chunyang Li
648be739152fc5a80d2c7c93
Zheyuan Zhang
claimed_verified
2024-06-28T08:03:43.894Z
648c4af819bb04c06467189c
/avatars/8b9372a233d4c00b555625fa7b5203e2.svg
Zheyuan Zhang
user
Zheyuan22
648be739152fc5a80d2c7c94
Yushi Bai
claimed_verified
2024-02-01T08:45:17.984Z
64ed568ccf6118a9379a61b8
/avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
Yushi Bai
user
bys0318
648be739152fc5a80d2c7c95
Yantao Liu
claimed_verified
2023-06-16T17:38:29.470Z
62e25e2247678ea5ce1b1786
/avatars/1bb32e7597a9b1c89c434cbf550b5382.svg
Yantao
user
RicardoL1u
648be739152fc5a80d2c7c96
Amy Xin
claimed_verified
2024-07-09T08:17:47.017Z
660bf98c3336a7e128a0e918
/avatars/3e3f2886bd4a730ec19b13aecc99279f.svg
Amy Xin
user
amyxx2001
648be739152fc5a80d2c7c97
Nianyi Lin
648be739152fc5a80d2c7c98
Kaifeng Yun
648be739152fc5a80d2c7c99
Linlu Gong
648be739152fc5a80d2c7c9a
Jianhui Chen
648be739152fc5a80d2c7c9b
Zhili Wu
648be739152fc5a80d2c7c9c
Yunjia Qi
648be739152fc5a80d2c7c9d
Weikai Li
648be739152fc5a80d2c7c9e
Yong Guan
648be739152fc5a80d2c7c9f
Kaisheng Zeng
648be739152fc5a80d2c7ca0
Ji Qi
claimed_verified
2025-07-02T08:24:25.122Z
6864b9f90a269bace8c92164
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zSeWHZRCZtzZ13CzFrr5u.png
Ji Qi
user
miracle11121
648be739152fc5a80d2c7ca1
Hailong Jin
648be739152fc5a80d2c7ca2
Jinxin Liu
648be739152fc5a80d2c7ca3
Yu Gu
claimed_verified
2024-10-14T14:08:46.423Z
5e7e595230dc073f817a2bb5
/avatars/d5ff36e45555d9e169cf56c845736444.svg
Yu Gu
user
entslscheia
648be739152fc5a80d2c7ca4
Yuan Yao
648be739152fc5a80d2c7ca5
Ning Ding
claimed_verified
2025-01-31T09:51:06.866Z
60cf4bcb1ce3775ebb86e5d5
/avatars/12bcd18d215abf91f297f93007733148.svg
Ning Ding
user
stingning
648be739152fc5a80d2c7ca6
Lei Hou
648be739152fc5a80d2c7ca7
Zhiyuan Liu
claimed_verified
2025-06-02T08:02:54.391Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
648be739152fc5a80d2c7ca8
Bin Xu
648be739152fc5a80d2c7ca9
Jie Tang
648be739152fc5a80d2c7caa
Juanzi Li
KoLA: Carefully Benchmarking World Knowledge of Large Language Models
The unprecedented performance of large language models (LLMs) necessitates
improvements in evaluations. Rather than merely exploring the breadth of LLM
abilities, we believe meticulous and thoughtful designs are essential to
thorough, unbiased, and applicable evaluations. Given the importance of world
knowledge to LLMs, we construct a Knowledge-oriented LLM Assessment benchmark
(KoLA), in which we carefully design three crucial factors: (1) For ability
modeling, we mimic human cognition to form a four-level taxonomy of
knowledge-related abilities, covering 19 tasks. (2) For data, to ensure fair
comparisons, we use both Wikipedia, a corpus prevalently pre-trained by LLMs,
along with continuously collected emerging corpora, aiming to evaluate the
capacity to handle unseen data and evolving knowledge. (3) For evaluation
criteria, we adopt a contrastive system, including overall standard scores for
better numerical comparability across tasks and models and a unique
self-contrast metric for automatically evaluating knowledge hallucination. We
evaluate 21 open-source and commercial LLMs and obtain some intriguing
findings. The KoLA dataset and open-participation leaderboard are publicly
released at https://kola.xlore.cn and will be continuously updated to provide
references for developing LLMs and knowledge-related systems.
A benchmark for evaluating large language models emphasizes knowledge-related abilities, uses a mix of datasets, and incorporates self-contrast metrics to detect knowledge hallucination.
large language models (LLMs) Knowledge-oriented LLM Assessment benchmark (KoLA) four-level taxonomy knowledge-related abilities Wikipedia emerging corpora contrastive system self-contrast metric knowledge hallucination