LongBench v2: Towards Deeper Understanding and Reasoning on Realistic
  Long-context Multitasks
This paper introduces LongBench v2, a benchmark designed to assess the
ability of LLMs to handle long-context problems requiring deep understanding
and reasoning across real-world multitasks. LongBench v2 consists of 503
challenging multiple-choice questions, with contexts ranging from 8k to 2M
words, across six major task categories: single-document QA, multi-document QA,
long in-context learning, long-dialogue history understanding, code repository
understanding, and long structured data understanding. To ensure the breadth
and the practicality, we collect data from nearly 100 highly educated
individuals with diverse professional backgrounds. We employ both automated and
manual review processes to maintain high quality and difficulty, resulting in
human experts achieving only 53.7% accuracy under a 15-minute time constraint.
Our evaluation reveals that the best-performing model, when directly answers
the questions, achieves only 50.1% accuracy. In contrast, the o1-preview model,
which includes longer reasoning, achieves 57.7%, surpassing the human baseline
by 4%. These results highlight the importance of enhanced reasoning ability and
scaling inference-time compute to tackle the long-context challenges in
LongBench v2. The project is available at https://longbench2.github.io.
6764d4d46be739a319253381
Yushi Bai
claimed_verified
2024-12-20T08:33:05.814Z
64ed568ccf6118a9379a61b8
/avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
Yushi Bai
user
bys0318
6764d4d46be739a319253382
Shangqing Tu
claimed_verified
2024-12-20T08:33:03.700Z
648c48d8c0ddeee6df5b6d22
/avatars/8706b0b16dfc332b96c91d3ced31bd0b.svg
Shangqing Tu
user
tsq2000
6764d4d46be739a319253383
Jiajie Zhang
admin_assigned
2024-12-20T09:20:03.123Z
66cdd285c51a915bd5f2d017
/avatars/14e5794307e4672b1b51d26b31227e0f.svg
Jiajie Zhang
user
NeoZ123
6764d4d46be739a319253384
Hao Peng
admin_assigned
2024-12-20T09:19:34.830Z
660ec5a2509153ca49775a7c
/avatars/97570fc245cc8ec7628da9c13bd35b71.svg
Hao Peng
user
haopeng01
6764d4d46be739a319253385
Xiaozhi Wang
admin_assigned
2024-12-20T09:19:21.316Z
648c48e7a47a0850f9458b67
/avatars/47d71d80f9901313feb0199c37296389.svg
Xiaozhi Wang
user
wangxz098
6764d4d46be739a319253386
Xin Lv
6764d4d46be739a319253387
Shulin Cao
admin_assigned
2024-12-20T09:19:04.990Z
62c924a6334a6ee11c2e8dfa
/avatars/3dbc37af162b94d68cb83665ac4528c3.svg
ShulinCao
user
caoshulin
6764d4d46be739a319253388
Jiazheng Xu
admin_assigned
2024-12-20T09:18:59.503Z
62d7b131f6e8ba66107af761
/avatars/f1c5df47aef69c824fd166722df8f670.svg
Jiazheng Xu
user
xujz0703
6764d4d46be739a319253389
Lei Hou
6764d4d46be739a31925338a
Yuxiao Dong
admin_assigned
2024-12-20T09:18:30.529Z
640e73bdfdeaae1390857b62
/avatars/cd6779e30f716002a7838ed93d5c0754.svg
Yuxiao Dong
user
yuxiaod
6764d4d46be739a31925338b
Jie Tang
admin_assigned
2024-12-20T09:18:23.208Z
640dff05474aa6f89556677e
/avatars/1b4591c7322d649c797b3125148f1915.svg
Jie Tang
user
jerytang
6764d4d46be739a31925338c
Juanzi Li
admin_assigned
2024-12-20T09:18:13.826Z
65df8cbc2705d9672f55d1aa
/avatars/63e46f15bb76bd9d4508fd0f54f39829.svg
Juanzi Li
user
juanli
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic
  Long-context Multitasks
This paper introduces LongBench v2, a benchmark designed to assess the
ability of LLMs to handle long-context problems requiring deep understanding
and reasoning across real-world multitasks. LongBench v2 consists of 503
challenging multiple-choice questions, with contexts ranging from 8k to 2M
words, across six major task categories: single-document QA, multi-document QA,
long in-context learning, long-dialogue history understanding, code repository
understanding, and long structured data understanding. To ensure the breadth
and the practicality, we collect data from nearly 100 highly educated
individuals with diverse professional backgrounds. We employ both automated and
manual review processes to maintain high quality and difficulty, resulting in
human experts achieving only 53.7% accuracy under a 15-minute time constraint.
Our evaluation reveals that the best-performing model, when directly answers
the questions, achieves only 50.1% accuracy. In contrast, the o1-preview model,
which includes longer reasoning, achieves 57.7%, surpassing the human baseline
by 4%. These results highlight the importance of enhanced reasoning ability and
scaling inference-time compute to tackle the long-context challenges in
LongBench v2. The project is available at https://longbench2.github.io.
A benchmark evaluates large language models' performance on long-context problems across various tasks, showing that enhanced reasoning and increased computational resources improve accuracy beyond human performance.
LLMs multiple-choice questions single-document QA multi-document QA long in-context learning long-dialogue history understanding code repository understanding long structured data understanding human experts automated review manual review reasoning ability inference-time compute o1-preview model