Why Do Transformers Fail to Forecast Time Series In-Context?
Time series forecasting (TSF) remains a challenging and largely unsolved
problem in machine learning, despite significant recent efforts leveraging
Large Language Models (LLMs), which predominantly rely on Transformer
architectures. Empirical evidence consistently shows that even powerful
Transformers often fail to outperform much simpler models, e.g., linear models,
on TSF tasks; however, a rigorous theoretical understanding of this phenomenon
remains limited. In this paper, we provide a theoretical analysis of
Transformers' limitations for TSF through the lens of In-Context Learning (ICL)
theory. Specifically, under AR(p) data, we establish that: (1) Linear
Self-Attention (LSA) models cannot achieve lower expected MSE than
classical linear models for in-context forecasting; (2) as the context length
approaches to infinity, LSA asymptotically recovers the optimal linear
predictor; and (3) under Chain-of-Thought (CoT) style inference, predictions
collapse to the mean exponentially. We empirically validate these findings
through carefully designed experiments. Our theory not only sheds light on
several previously underexplored phenomena but also offers practical insights
for designing more effective forecasting architectures. We hope our work
encourages the broader research community to revisit the fundamental
theoretical limitations of TSF and to critically evaluate the direct
application of increasingly sophisticated architectures without deeper
scrutiny.
68eff24136f8b025381e17fb
Yufa Zhou
claimed_verified
2025-10-16T06:41:05.535Z
658ab894c4b2004663dff3ae
https://cdn-avatars.huggingface.co/v1/production/uploads/658ab894c4b2004663dff3ae/oPRnFuW2Imaa2KkYWNbSf.jpeg
YUFA ZHOU
user
MasterZhou
68eff24136f8b025381e17fc
Yixiao Wang
68eff24136f8b025381e17fd
Surbhi Goel
68eff24136f8b025381e17fe
Anru R. Zhang
Why Do Transformers Fail to Forecast Time Series In-Context?
Time series forecasting (TSF) remains a challenging and largely unsolved
problem in machine learning, despite significant recent efforts leveraging
Large Language Models (LLMs), which predominantly rely on Transformer
architectures. Empirical evidence consistently shows that even powerful
Transformers often fail to outperform much simpler models, e.g., linear models,
on TSF tasks; however, a rigorous theoretical understanding of this phenomenon
remains limited. In this paper, we provide a theoretical analysis of
Transformers' limitations for TSF through the lens of In-Context Learning (ICL)
theory. Specifically, under AR(p) data, we establish that: (1) Linear
Self-Attention (LSA) models cannot achieve lower expected MSE than
classical linear models for in-context forecasting; (2) as the context length
approaches to infinity, LSA asymptotically recovers the optimal linear
predictor; and (3) under Chain-of-Thought (CoT) style inference, predictions
collapse to the mean exponentially. We empirically validate these findings
through carefully designed experiments. Our theory not only sheds light on
several previously underexplored phenomena but also offers practical insights
for designing more effective forecasting architectures. We hope our work
encourages the broader research community to revisit the fundamental
theoretical limitations of TSF and to critically evaluate the direct
application of increasingly sophisticated architectures without deeper
scrutiny.
Theoretical analysis reveals that Transformers, particularly Linear Self-Attention models, have limitations in time series forecasting compared to classical linear models, with predictions collapsing to the mean under Chain-of-Thought inference.
Transformers In-Context Learning Linear Self-Attention AR($p$) data expected MSE classical linear models Chain-of-Thought inference