PRELUDE: A Benchmark Designed to Require Global Comprehension and
  Reasoning over Long Contexts
We introduce PRELUDE, a benchmark for evaluating long-context understanding
through the task of determining whether a character's prequel story is
consistent with the canonical narrative of the original book. Our task poses a
stronger demand for global comprehension and deep reasoning than existing
benchmarks -- as the prequels are not part of the original story, assessing
their plausibility typically requires searching and integrating information
that is only indirectly related. Empirically, 88% of instances require evidence
from multiple parts of the narrative. Experimental results highlight the
challenge of our task: in-context learning, RAG and in-domain training with
state-of-the-art LLMs, and commercial DeepResearch services, lag behind humans
by >15%. A further human study reveals that models often produce correct
answers with flawed reasoning, leading to an over 30% gap in reasoning accuracy
compared to humans. These findings underscore the substantial room for
improvement in long-context understanding and reasoning.
689dc868b083e610d741eb30
Mo Yu
claimed_verified
2025-08-20T08:53:21.358Z
67af92045a86287292026808
/avatars/8bad9272fe73ba04e077b5484837c8d3.svg
Mo
user
BishopGorov
689dc868b083e610d741eb31
Tsz Ting Chung
689dc868b083e610d741eb32
Chulun Zhou
689dc868b083e610d741eb33
Tong Li
689dc868b083e610d741eb34
Rui Lu
689dc868b083e610d741eb35
Jiangnan Li
689dc868b083e610d741eb36
Liyan Xu
claimed_verified
2025-08-19T10:50:01.477Z
650f0fac11f3210cf7a8a849
/avatars/687d56c3a6d4f5cdb34e424cdcff954d.svg
Liyan Xu
user
lxucs
689dc868b083e610d741eb37
Haoshu Lu
689dc868b083e610d741eb38
Ning Zhang
689dc868b083e610d741eb39
Jing Li
689dc868b083e610d741eb3a
Jie Zhou
PRELUDE: A Benchmark Designed to Require Global Comprehension and
  Reasoning over Long Contexts
We introduce PRELUDE, a benchmark for evaluating long-context understanding
through the task of determining whether a character's prequel story is
consistent with the canonical narrative of the original book. Our task poses a
stronger demand for global comprehension and deep reasoning than existing
benchmarks -- as the prequels are not part of the original story, assessing
their plausibility typically requires searching and integrating information
that is only indirectly related. Empirically, 88% of instances require evidence
from multiple parts of the narrative. Experimental results highlight the
challenge of our task: in-context learning, RAG and in-domain training with
state-of-the-art LLMs, and commercial DeepResearch services, lag behind humans
by >15%. A further human study reveals that models often produce correct
answers with flawed reasoning, leading to an over 30% gap in reasoning accuracy
compared to humans. These findings underscore the substantial room for
improvement in long-context understanding and reasoning.
A benchmark called PRELUDE evaluates long-context understanding by assessing the consistency of prequel stories with original books, revealing significant challenges for models compared to humans.
PRELUDE long-context understanding in-context learning RAG state-of-the-art LLMs DeepResearch reasoning accuracy