A Controllable Examination for Long-Context Language Models
Existing frameworks for evaluating long-context language models (LCLM) can be
broadly categorized into real-world and synthetic tasks. Despite their utility,
both approaches are accompanied by certain intrinsic limitations. Real-world
tasks are too complex to interpret or characterize and are susceptible to data
contamination. In contrast, synthetic tasks often adopt the
needle-in-the-haystack (NIAH) format, wherein a lack of coherence between the
"needle" and the "haystack" compromises their validity as proxies for realistic
applications. In response to these challenges, we posit that an ideal
long-context evaluation framework should be characterized by three essential
features: seamless context, controllable setting, and
sound evaluation. This study introduces LongBioBench, a
novel benchmark that utilizes artificially generated biographies as a
controlled environment for assessing LCLMs across dimensions of
understanding, reasoning, and trustworthiness.
Our experimental evaluation, which includes 18 LCLMs in total,
demonstrates that most models still exhibit deficiencies in semantic
understanding and elementary reasoning over retrieved results and are less
trustworthy as context length increases. Our further analysis indicates some
design choices employed by existing synthetic benchmarks, such as contextual
non-coherence, numerical needles, and the absence of distractors, rendering
them vulnerable to test the model long-context capabilities. Moreover, we also
reveal that long-context continual pretraining primarily adjusts RoPE embedding
to accommodate extended context lengths. To sum up, compared to previous
synthetic benchmarks, LongBioBench achieves a better trade-off between
mirroring authentic language tasks and maintaining controllability, and is
highly interpretable and configurable.
683ff4dcfbc9041ef7274c51
Yijun Yang
claimed_verified
2025-06-04T08:53:47.455Z
657eea68f4f72f2c4c44640d
/avatars/033bc4f063cd36a79a0b4761f6ebe32c.svg
Yijun YANG
user
thomasyyj
683ff4dcfbc9041ef7274c52
Zeyu Huang
683ff4dcfbc9041ef7274c53
Wenhao Zhu
admin_assigned
2025-06-05T12:47:37.569Z
649d7d8968586ca9bf7f5fe6
/avatars/b444240770d4025dea41871cf38126dc.svg
Wenhao Zhu
user
Wenhao97
683ff4dcfbc9041ef7274c54
Zihan Qiu
admin_assigned
2025-06-05T12:47:15.684Z
647ccbd6e07cf9bb2d485244
/avatars/e8915abaff04f6762247e196b7cf84df.svg
Zihan Qiu
user
QwQZh
683ff4dcfbc9041ef7274c55
Fei Yuan
683ff4dcfbc9041ef7274c56
Jeff Z. Pan
683ff4dcfbc9041ef7274c57
Ivan Titov
admin_assigned
2025-06-05T12:46:49.924Z
666f6375623133f1ce79c021
/avatars/b57a457cff81d685fdd9eadee53445fc.svg
Ivan Titov
user
Ivanchoo
A Controllable Examination for Long-Context Language Models
Existing frameworks for evaluating long-context language models (LCLM) can be
broadly categorized into real-world and synthetic tasks. Despite their utility,
both approaches are accompanied by certain intrinsic limitations. Real-world
tasks are too complex to interpret or characterize and are susceptible to data
contamination. In contrast, synthetic tasks often adopt the
needle-in-the-haystack (NIAH) format, wherein a lack of coherence between the
"needle" and the "haystack" compromises their validity as proxies for realistic
applications. In response to these challenges, we posit that an ideal
long-context evaluation framework should be characterized by three essential
features: seamless context, controllable setting, and
sound evaluation. This study introduces LongBioBench, a
novel benchmark that utilizes artificially generated biographies as a
controlled environment for assessing LCLMs across dimensions of
understanding, reasoning, and trustworthiness.
Our experimental evaluation, which includes 18 LCLMs in total,
demonstrates that most models still exhibit deficiencies in semantic
understanding and elementary reasoning over retrieved results and are less
trustworthy as context length increases. Our further analysis indicates some
design choices employed by existing synthetic benchmarks, such as contextual
non-coherence, numerical needles, and the absence of distractors, rendering
them vulnerable to test the model long-context capabilities. Moreover, we also
reveal that long-context continual pretraining primarily adjusts RoPE embedding
to accommodate extended context lengths. To sum up, compared to previous
synthetic benchmarks, LongBioBench achieves a better trade-off between
mirroring authentic language tasks and maintaining controllability, and is
highly interpretable and configurable.
LongBioBench is a new benchmark using artificially generated biographies to evaluate long-context language models across understanding, reasoning, and trustworthiness dimensions, addressing limitations in existing frameworks.
long-context language models (LCLM) real-world tasks synthetic tasks needle-in-the-haystack (NIAH) seamless context controllable setting sound evaluation LongBioBench semantic understanding elementary reasoning trustworthiness long-context continual pretraining RoPE embedding