ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via
  Chart-to-Code Generation
We introduce a new benchmark, ChartMimic, aimed at assessing the
visually-grounded code generation capabilities of large multimodal models
(LMMs). ChartMimic utilizes information-intensive visual charts and textual
instructions as inputs, requiring LMMs to generate the corresponding code for
chart rendering. ChartMimic includes 1,000 human-curated (figure, instruction,
code) triplets, which represent the authentic chart use cases found in
scientific papers across various domains(e.g., Physics, Computer Science,
Economics, etc). These charts span 18 regular types and 4 advanced types,
diversifying into 191 subcategories. Furthermore, we propose multi-level
evaluation metrics to provide an automatic and thorough assessment of the
output code and the rendered charts. Unlike existing code generation
benchmarks, ChartMimic places emphasis on evaluating LMMs' capacity to
harmonize a blend of cognitive capabilities, encompassing visual understanding,
code generation, and cross-modal reasoning. The evaluation of 3 proprietary
models and 11 open-weight models highlights the substantial challenges posed by
ChartMimic. Even the advanced GPT-4V, Claude-3-opus only achieve an average
score of 73.2 and 53.7, respectively, indicating significant room for
improvement. We anticipate that ChartMimic will inspire the development of
LMMs, advancing the pursuit of artificial general intelligence.
666fdf3bc16d62cbb045cae0
Chufan Shi
admin_assigned
2024-06-17T14:01:12.424Z
6373b559c3b104ef179cf503
/avatars/36f36eea72a100ee56b37c6d185b47c0.svg
Shi
user
Chufan
666fdf3bc16d62cbb045cae1
Cheng Yang
claimed_verified
2024-06-17T07:31:03.951Z
64b75db417570fdff9b43fd9
https://cdn-avatars.huggingface.co/v1/production/uploads/64b75db417570fdff9b43fd9/-NcNL_QzuiAAvsBIQG3xo.png
Cheng YANG
user
Cheng-YANG
666fdf3bc16d62cbb045cae2
Yaxin Liu
666fdf3bc16d62cbb045cae3
Bo Shui
666fdf3bc16d62cbb045cae4
Junjie Wang
claimed_verified
2024-06-18T07:20:00.036Z
62579c55b98dcaa7e0de285d
https://cdn-avatars.huggingface.co/v1/production/uploads/62579c55b98dcaa7e0de285d/0YUd5nloul_bW9yolDGGo.jpeg
wangjunjie
user
wanng
666fdf3bc16d62cbb045cae5
Mohan Jing
666fdf3bc16d62cbb045cae6
Linran Xu
666fdf3bc16d62cbb045cae7
Xinyu Zhu
claimed_verified
2024-06-19T20:21:19.706Z
623b290048f658f28aef79f7
https://cdn-avatars.huggingface.co/v1/production/uploads/1648044277149-noauth.jpeg
Xinyu Zhu
user
TianHongZXY
666fdf3bc16d62cbb045cae8
Siheng Li
admin_assigned
2024-06-17T14:09:59.455Z
62f71b23d3bdacb7eec43b1f
https://cdn-avatars.huggingface.co/v1/production/uploads/62f71b23d3bdacb7eec43b1f/6RNVG9zug9tutjo5i8T4C.jpeg
SihengLi
user
SihengLi
666fdf3bc16d62cbb045cae9
Yuxiang Zhang
claimed_verified
2024-07-22T07:12:32.284Z
6239386535384c2bcccd2a4f
/avatars/c62cefc6235210ca12b46beb62751d8b.svg
Yuxiang Zhang
user
Joelzhang
666fdf3bc16d62cbb045caea
Gongye Liu
admin_assigned
2024-06-17T14:10:43.252Z
6553316bf151de82f6a23e1d
https://cdn-avatars.huggingface.co/v1/production/uploads/6553316bf151de82f6a23e1d/GTBkSj4Fa3OoyM6Muz_Sc.jpeg
Gongye Liu
user
liuhuohuo
666fdf3bc16d62cbb045caeb
Xiaomei Nie
666fdf3bc16d62cbb045caec
Deng Cai
claimed_verified
2024-06-20T07:34:42.506Z
646735a1946476c5d214e5b2
/avatars/23392fda23fa6288fcb309f7eaa73472.svg
Deng Cai
user
jcyk
666fdf3bc16d62cbb045caed
Yujiu Yang
admin_assigned
2024-06-17T14:11:00.402Z
64ca1fe838837b12d5e529b7
/avatars/44a3ad9e59318784ac531993b5f69f6b.svg
Yujiu Yang
user
Thu-redrobot
ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via
  Chart-to-Code Generation
We introduce a new benchmark, ChartMimic, aimed at assessing the
visually-grounded code generation capabilities of large multimodal models
(LMMs). ChartMimic utilizes information-intensive visual charts and textual
instructions as inputs, requiring LMMs to generate the corresponding code for
chart rendering. ChartMimic includes 1,000 human-curated (figure, instruction,
code) triplets, which represent the authentic chart use cases found in
scientific papers across various domains(e.g., Physics, Computer Science,
Economics, etc). These charts span 18 regular types and 4 advanced types,
diversifying into 191 subcategories. Furthermore, we propose multi-level
evaluation metrics to provide an automatic and thorough assessment of the
output code and the rendered charts. Unlike existing code generation
benchmarks, ChartMimic places emphasis on evaluating LMMs' capacity to
harmonize a blend of cognitive capabilities, encompassing visual understanding,
code generation, and cross-modal reasoning. The evaluation of 3 proprietary
models and 11 open-weight models highlights the substantial challenges posed by
ChartMimic. Even the advanced GPT-4V, Claude-3-opus only achieve an average
score of 73.2 and 53.7, respectively, indicating significant room for
improvement. We anticipate that ChartMimic will inspire the development of
LMMs, advancing the pursuit of artificial general intelligence.
ChartMimic is a new benchmark assessing large multimodal models' visually-grounded code generation for chart rendering, evaluating multi-level cognitive capabilities with diverse chart types and metrics.
multimodal models visually-grounded code generation visual charts textual instructions multi-level evaluation metrics cognitive capabilities visual understanding cross-modal reasoning