ChartMuseum: Testing Visual Reasoning Capabilities of Large
  Vision-Language Models
Chart understanding presents a unique challenge for large vision-language
models (LVLMs), as it requires the integration of sophisticated textual and
visual reasoning capabilities. However, current LVLMs exhibit a notable
imbalance between these skills, falling short on visual reasoning that is
difficult to perform in text. We conduct a case study using a synthetic dataset
solvable only through visual reasoning and show that model performance degrades
significantly with increasing visual complexity, while human performance
remains robust. We then introduce ChartMuseum, a new Chart Question Answering
(QA) benchmark containing 1,162 expert-annotated questions spanning multiple
reasoning types, curated from real-world charts across 184 sources,
specifically built to evaluate complex visual and textual reasoning. Unlike
prior chart understanding benchmarks -- where frontier models perform similarly
and near saturation -- our benchmark exposes a substantial gap between model
and human performance, while effectively differentiating model capabilities:
although humans achieve 93% accuracy, the best-performing model Gemini-2.5-Pro
attains only 63.0%, and the leading open-source LVLM Qwen2.5-VL-72B-Instruct
achieves only 38.5%. Moreover, on questions requiring primarily visual
reasoning, all models experience a 35%-55% performance drop from
text-reasoning-heavy question performance. Lastly, our qualitative error
analysis reveals specific categories of visual reasoning that are challenging
for current LVLMs.
682bf33a6f59c839338ffdd0
Liyan Tang
claimed_verified
2025-05-20T07:20:38.292Z
62c70672e7d825deaae41e5e
https://cdn-avatars.huggingface.co/v1/production/uploads/62c70672e7d825deaae41e5e/ICCpeBwmQ1NsgWcjG-MEZ.png
Liyan Tang
user
lytang
682bf33a6f59c839338ffdd1
Grace Kim
682bf33a6f59c839338ffdd2
Xinyu Zhao
claimed_verified
2025-05-21T09:03:40.939Z
65f356f3244d1b4066bcc62b
/avatars/9eedf36a95e6d573bd3f8c0f0d16a9d6.svg
Xinyu (Lucy) Zhao
user
xyz-zy
682bf33a6f59c839338ffdd3
Thom Lake
admin_assigned
2025-05-20T08:43:44.062Z
64a87c60b76bfd863e715cab
https://cdn-avatars.huggingface.co/v1/production/uploads/64a87c60b76bfd863e715cab/cpAUTOTEwhgP29aw6AOWA.jpeg
Thom Lake
user
thomlake
682bf33a6f59c839338ffdd4
Wenxuan Ding
claimed_verified
2025-05-20T19:29:36.814Z
6502b65106c324fee1089ad5
/avatars/9e054900bc1dd9a178a427ca9cf939da.svg
Wenxuan D
user
wenwenD
682bf33a6f59c839338ffdd5
Fangcong Yin
admin_assigned
2025-05-20T08:43:29.872Z
64efa8748602335a044cd97f
/avatars/0ab5df922cb0ce4abe7aed35e7b9100c.svg
Fangcong Yin
user
fcyin
682bf33a6f59c839338ffdd6
Prasann Singhal
admin_assigned
2025-05-20T08:43:24.263Z
613cb3e1c7a43c281cd417a2
/avatars/69123ba49c2aa1cd9f3cc5746f4839dc.svg
Prasann Singhal
user
PrasannSinghal
682bf33a6f59c839338ffdd7
Manya Wadhwa
admin_assigned
2025-05-20T08:43:18.026Z
655ab2ccc11dee7f7e6db119
/avatars/1c13e338cd4cc0b4eb681ed8f33abf19.svg
Manya Wadhwa
user
wadhma
682bf33a6f59c839338ffdd8
Zeyu Leo Liu
admin_assigned
2025-05-20T08:42:57.568Z
6607b0d29d2edd43f74dec98
/avatars/437b5cbc555bf6906c3f07495a903ab4.svg
Zeyu Leo Liu
user
leo-liuzy
682bf33a6f59c839338ffdd9
Zayne Sprague
admin_assigned
2025-05-20T08:42:49.622Z
64e78a03e3953cd90bcad620
https://cdn-avatars.huggingface.co/v1/production/uploads/64e78a03e3953cd90bcad620/Rj_-xLJUsxdRmNhvRbssq.jpeg
Zayne Sprague
user
Zaynes
682bf33a6f59c839338ffdda
Ramya Namuduri
682bf33a6f59c839338ffddb
Bodun Hu
682bf33a6f59c839338ffddc
Juan Diego Rodriguez
682bf33a6f59c839338ffddd
Puyuan Peng
admin_assigned
2025-05-20T08:42:13.748Z
6480706f5409aa3e3bbaee16
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/qi2IrGGu7rgQVB_cfPNhh.png
Puyuan Peng
user
pyp1
682bf33a6f59c839338ffdde
Greg Durrett
admin_assigned
2025-05-20T08:42:07.638Z
65be9918b54ab5b37d1b67a7
/avatars/9953707affb6881724c8efb2abf0c668.svg
Greg Durrett
user
gregdurrett
ChartMuseum: Testing Visual Reasoning Capabilities of Large
  Vision-Language Models
Chart understanding presents a unique challenge for large vision-language
models (LVLMs), as it requires the integration of sophisticated textual and
visual reasoning capabilities. However, current LVLMs exhibit a notable
imbalance between these skills, falling short on visual reasoning that is
difficult to perform in text. We conduct a case study using a synthetic dataset
solvable only through visual reasoning and show that model performance degrades
significantly with increasing visual complexity, while human performance
remains robust. We then introduce ChartMuseum, a new Chart Question Answering
(QA) benchmark containing 1,162 expert-annotated questions spanning multiple
reasoning types, curated from real-world charts across 184 sources,
specifically built to evaluate complex visual and textual reasoning. Unlike
prior chart understanding benchmarks -- where frontier models perform similarly
and near saturation -- our benchmark exposes a substantial gap between model
and human performance, while effectively differentiating model capabilities:
although humans achieve 93% accuracy, the best-performing model Gemini-2.5-Pro
attains only 63.0%, and the leading open-source LVLM Qwen2.5-VL-72B-Instruct
achieves only 38.5%. Moreover, on questions requiring primarily visual
reasoning, all models experience a 35%-55% performance drop from
text-reasoning-heavy question performance. Lastly, our qualitative error
analysis reveals specific categories of visual reasoning that are challenging
for current LVLMs.
A new benchmark, ChartMuseum, highlights the underperformance of large vision-language models in chart question answering, particularly for visually complex questions, compared to human accuracy.
vision-language models LVLMs visual reasoning textual reasoning synthetic dataset ChartMuseum Chart Question Answering QA benchmark reasoning types Gemini-2.5-Pro Qwen2.5-VL-72B-Instruct qualitative error analysis