Pitfalls in Evaluating Language Model Forecasters
Large language models (LLMs) have recently been applied to forecasting tasks,
with some works claiming these systems match or exceed human performance. In
this paper, we argue that, as a community, we should be careful about such
conclusions as evaluating LLM forecasters presents unique challenges. We
identify two broad categories of issues: (1) difficulty in trusting evaluation
results due to many forms of temporal leakage, and (2) difficulty in
extrapolating from evaluation performance to real-world forecasting. Through
systematic analysis and concrete examples from prior work, we demonstrate how
evaluation flaws can raise concerns about current and future performance
claims. We argue that more rigorous evaluation methodologies are needed to
confidently assess the forecasting abilities of LLMs.
683ee251668eea64b040a050
Daniel Paleka
admin_assigned
2025-06-03T12:43:51.277Z
62363277cd8e1f47c865190a
/avatars/fb42d42dedd4dde8df4109883cbe7ea9.svg
Daniel Paleka
user
dpaleka
683ee251668eea64b040a051
Shashwat Goel
683ee251668eea64b040a052
Jonas Geiping
admin_assigned
2025-06-03T12:44:04.836Z
63d86dbf3130cadcaf8bdd11
/avatars/29d79a0c6dcec01111ef192fecd0fa7a.svg
Jonas Geiping
user
JonasGeiping
683ee251668eea64b040a053
Florian Tramèr
admin_assigned
2025-06-03T12:44:13.091Z
63568f18ba90b4ea9fe91cb5
/avatars/3e8b3c573e20cf80d329a312bfc34728.svg
Florian Tramer
user
ftramer
Pitfalls in Evaluating Language Model Forecasters
Large language models (LLMs) have recently been applied to forecasting tasks,
with some works claiming these systems match or exceed human performance. In
this paper, we argue that, as a community, we should be careful about such
conclusions as evaluating LLM forecasters presents unique challenges. We
identify two broad categories of issues: (1) difficulty in trusting evaluation
results due to many forms of temporal leakage, and (2) difficulty in
extrapolating from evaluation performance to real-world forecasting. Through
systematic analysis and concrete examples from prior work, we demonstrate how
evaluation flaws can raise concerns about current and future performance
claims. We argue that more rigorous evaluation methodologies are needed to
confidently assess the forecasting abilities of LLMs.
Evaluation of large language models for forecasting is flawed due to temporal leakage and difficulties in translating performance to real-world forecasting, necessitating more rigorous methodologies.
large language models forecasting temporal leakage evaluation methodologies