ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in
  Complex Chart Question Answering
Recent multimodal LLMs have shown promise in chart-based visual question
answering, but their performance declines sharply on unannotated charts, those
requiring precise visual interpretation rather than relying on textual
shortcuts. To address this, we introduce ChartAgent, a novel agentic framework
that explicitly performs visual reasoning directly within the chart's spatial
domain. Unlike textual chain-of-thought reasoning, ChartAgent iteratively
decomposes queries into visual subtasks and actively manipulates and interacts
with chart images through specialized actions such as drawing annotations,
cropping regions (e.g., segmenting pie slices, isolating bars), and localizing
axes, using a library of chart-specific vision tools to fulfill each subtask.
This iterative reasoning process closely mirrors human cognitive strategies for
chart comprehension. ChartAgent achieves state-of-the-art accuracy on the
ChartBench and ChartX benchmarks, surpassing prior methods by up to 16.07%
absolute gain overall and 17.31% on unannotated, numerically intensive queries.
Furthermore, our analyses show that ChartAgent is (a) effective across diverse
chart types, (b) achieve the highest scores across varying visual and reasoning
complexity levels, and (c) serves as a plug-and-play framework that boosts
performance across diverse underlying LLMs. Our work is among the first to
demonstrate visually grounded reasoning for chart understanding using
tool-augmented multimodal agents.
68e51f2a7e1f41b92bf8e9d7
Rachneet Kaur
claimed_verified
2025-10-08T08:01:32.102Z
663c57534edaac49a8afe286
https://cdn-avatars.huggingface.co/v1/production/uploads/663c57534edaac49a8afe286/fUB5uphwK-lqZE6EN2Z8F.jpeg
Rachneet Kaur
user
rachneetkaur
68e51f2a7e1f41b92bf8e9d8
Nishan Srishankar
68e51f2a7e1f41b92bf8e9d9
Zhen Zeng
68e51f2a7e1f41b92bf8e9da
Sumitra Ganesh
68e51f2a7e1f41b92bf8e9db
Manuela Veloso
ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in
  Complex Chart Question Answering
Recent multimodal LLMs have shown promise in chart-based visual question
answering, but their performance declines sharply on unannotated charts, those
requiring precise visual interpretation rather than relying on textual
shortcuts. To address this, we introduce ChartAgent, a novel agentic framework
that explicitly performs visual reasoning directly within the chart's spatial
domain. Unlike textual chain-of-thought reasoning, ChartAgent iteratively
decomposes queries into visual subtasks and actively manipulates and interacts
with chart images through specialized actions such as drawing annotations,
cropping regions (e.g., segmenting pie slices, isolating bars), and localizing
axes, using a library of chart-specific vision tools to fulfill each subtask.
This iterative reasoning process closely mirrors human cognitive strategies for
chart comprehension. ChartAgent achieves state-of-the-art accuracy on the
ChartBench and ChartX benchmarks, surpassing prior methods by up to 16.07%
absolute gain overall and 17.31% on unannotated, numerically intensive queries.
Furthermore, our analyses show that ChartAgent is (a) effective across diverse
chart types, (b) achieve the highest scores across varying visual and reasoning
complexity levels, and (c) serves as a plug-and-play framework that boosts
performance across diverse underlying LLMs. Our work is among the first to
demonstrate visually grounded reasoning for chart understanding using
tool-augmented multimodal agents.
ChartAgent, a novel agentic framework, performs visual reasoning directly within charts, achieving state-of-the-art accuracy on ChartBench and ChartX benchmarks by iteratively decomposing queries and using specialized visual actions.
multimodal LLMs visual question answering unannotated charts ChartAgent agentic framework visual reasoning spatial domain chain-of-thought reasoning visual subtasks drawing annotations cropping regions segmenting pie slices isolating bars localizing axes chart-specific vision tools ChartBench ChartX diverse chart types visual complexity reasoning complexity plug-and-play framework