MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning
  in LLMs
Recent large language models (LLMs) have demonstrated versatile capabilities
in long-context scenarios. Although some recent benchmarks have been developed
to evaluate the long-context capabilities of LLMs, there is a lack of
benchmarks evaluating the mathematical reasoning abilities of LLMs over long
contexts, which is crucial for LLMs' application in real-world scenarios. In
this paper, we introduce MathHay, an automated benchmark designed to assess the
long-context mathematical reasoning capabilities of LLMs. Unlike previous
benchmarks like Needle in a Haystack, which focus primarily on information
retrieval within long texts, MathHay demands models with both
information-seeking and complex mathematical reasoning abilities. We conduct
extensive experiments on MathHay to assess the long-context mathematical
reasoning abilities of eight top-performing LLMs. Even the best-performing
model, Gemini-1.5-Pro-002, still struggles with mathematical reasoning over
long contexts, achieving only 51.26% accuracy at 128K tokens. This highlights
the significant room for improvement on the MathHay benchmark.
6704d8f0e8d5909d35d32eea
Lei Wang
claimed_verified
2024-10-08T07:18:43.902Z
646def60df618b303b419323
/avatars/97aa761d5255abf230304cfeade87835.svg
Lei Wang
user
demolei
6704d8f0e8d5909d35d32eeb
Shan Dong
6704d8f0e8d5909d35d32eec
Yuhui Xu
admin_assigned
2024-10-08T09:34:06.566Z
6602869253a0518b2a98cafd
/avatars/c14b5953a716f42c83ad28147f8308ae.svg
Yuhui Xu
user
yuhuixu
6704d8f0e8d5909d35d32eed
Hanze Dong
admin_assigned
2024-10-08T09:33:48.011Z
63a3ff69f91ad3ea5703841d
/avatars/69227c4bce01d33747c1377b6f9672db.svg
Hanze Dong
user
hendrydong
6704d8f0e8d5909d35d32eee
Yalu Wang
admin_assigned
2024-10-08T09:33:41.725Z
656741fca637eb616aff92ba
/avatars/6a0c7ac44948406b191d27fd2157d829.svg
Yalu Wang
user
lunshi
6704d8f0e8d5909d35d32eef
Amrita Saha
6704d8f0e8d5909d35d32ef0
Ee-Peng Lim
6704d8f0e8d5909d35d32ef1
Caiming Xiong
admin_assigned
2024-10-08T09:33:30.216Z
649dbcc4e0fff1ed099dc80a
/avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
Caiming Xiong
user
cxiong
6704d8f0e8d5909d35d32ef2
Doyen Sahoo
admin_assigned
2024-10-08T09:33:24.218Z
65f84fd980481173afd91233
/avatars/6ac7bd6beba24d1476c5179b88c9e3fa.svg
Doyen
user
doyensahoo
MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning
  in LLMs
Recent large language models (LLMs) have demonstrated versatile capabilities
in long-context scenarios. Although some recent benchmarks have been developed
to evaluate the long-context capabilities of LLMs, there is a lack of
benchmarks evaluating the mathematical reasoning abilities of LLMs over long
contexts, which is crucial for LLMs' application in real-world scenarios. In
this paper, we introduce MathHay, an automated benchmark designed to assess the
long-context mathematical reasoning capabilities of LLMs. Unlike previous
benchmarks like Needle in a Haystack, which focus primarily on information
retrieval within long texts, MathHay demands models with both
information-seeking and complex mathematical reasoning abilities. We conduct
extensive experiments on MathHay to assess the long-context mathematical
reasoning abilities of eight top-performing LLMs. Even the best-performing
model, Gemini-1.5-Pro-002, still struggles with mathematical reasoning over
long contexts, achieving only 51.26% accuracy at 128K tokens. This highlights
the significant room for improvement on the MathHay benchmark.
MathHay is an automated benchmark assessing LLMs' long-context mathematical reasoning abilities, revealing significant areas for improvement.
large language models long-context mathematical reasoning MathHay information-seeking complex mathematical reasoning Gemini-1.5-Pro-002