When Benchmarks Age: Temporal Misalignment through Large Language Model
  Factuality Evaluation
The rapid evolution of large language models (LLMs) and the real world has
outpaced the static nature of widely used evaluation benchmarks, raising
concerns about their reliability for evaluating LLM factuality. While
substantial works continue to rely on the popular but old benchmarks, their
temporal misalignment with real-world facts and modern LLMs, and their effects
on LLM factuality evaluation remain underexplored. Therefore, in this work, we
present a systematic investigation of this issue by examining five popular
factuality benchmarks and eight LLMs released across different years. An
up-to-date fact retrieval pipeline and three metrics are tailored to quantify
benchmark aging and its impact on LLM factuality evaluation. Experimental
results and analysis illustrate that a considerable portion of samples in the
widely used factuality benchmarks are outdated, leading to unreliable
assessments of LLM factuality. We hope our work can provide a testbed to assess
the reliability of a benchmark for LLM factuality evaluation and inspire more
research on the benchmark aging issue. Codes are available in
https://github.com/JiangXunyi/BenchAge.
68e728707ae125f9582e69f4
Xunyi Jiang
claimed_verified
2025-10-09T06:50:03.984Z
645efecd7c6bff857734ceeb
/avatars/d1cda8b0b943165fb2de6769d28a6787.svg
XunyiJiang
user
Coooori
68e728707ae125f9582e69f5
Dingyi Chang
68e728707ae125f9582e69f6
Julian McAuley
68e728707ae125f9582e69f7
Xin Xu
claimed_verified
2025-10-09T06:50:06.149Z
6190ab805ca89a28e9f66873
/avatars/3c7ecc398fbf851acd2a132e947a92be.svg
Xin Xu
user
XinXuNLPer
When Benchmarks Age: Temporal Misalignment through Large Language Model
  Factuality Evaluation
The rapid evolution of large language models (LLMs) and the real world has
outpaced the static nature of widely used evaluation benchmarks, raising
concerns about their reliability for evaluating LLM factuality. While
substantial works continue to rely on the popular but old benchmarks, their
temporal misalignment with real-world facts and modern LLMs, and their effects
on LLM factuality evaluation remain underexplored. Therefore, in this work, we
present a systematic investigation of this issue by examining five popular
factuality benchmarks and eight LLMs released across different years. An
up-to-date fact retrieval pipeline and three metrics are tailored to quantify
benchmark aging and its impact on LLM factuality evaluation. Experimental
results and analysis illustrate that a considerable portion of samples in the
widely used factuality benchmarks are outdated, leading to unreliable
assessments of LLM factuality. We hope our work can provide a testbed to assess
the reliability of a benchmark for LLM factuality evaluation and inspire more
research on the benchmark aging issue. Codes are available in
https://github.com/JiangXunyi/BenchAge.
Research investigates the aging of factuality benchmarks and its impact on evaluating the factuality of large language models, revealing significant unreliability due to outdated samples.
large language models LLMs factuality benchmarks fact retrieval pipeline benchmark aging LLM factuality evaluation