Visual Programmability: A Guide for Code-as-Thought in Chart
  Understanding
Chart understanding presents a critical test to the reasoning capabilities of
Vision-Language Models (VLMs). Prior approaches face critical limitations: some
rely on external tools, making them brittle and constrained by a predefined
toolkit, while others fine-tune specialist models that often adopt a single
reasoning strategy, such as text-based chain-of-thought (CoT). The intermediate
steps of text-based reasoning are difficult to verify, which complicates the
use of reinforcement-learning signals that reward factual accuracy. To address
this, we propose a Code-as-Thought (CaT) approach to represent the visual
information of a chart in a verifiable, symbolic format. Our key insight is
that this strategy must be adaptive: a fixed, code-only implementation
consistently fails on complex charts where symbolic representation is
unsuitable. This finding leads us to introduce Visual Programmability: a
learnable property that determines if a chart-question pair is better solved
with code or direct visual analysis. We implement this concept in an adaptive
framework where a VLM learns to choose between the CaT pathway and a direct
visual reasoning pathway. The selection policy of the model is trained with
reinforcement learning using a novel dual-reward system. This system combines a
data-accuracy reward to ground the model in facts and prevent numerical
hallucination, with a decision reward that teaches the model when to use each
strategy, preventing it from defaulting to a single reasoning mode. Experiments
demonstrate strong and robust performance across diverse chart-understanding
benchmarks. Our work shows that VLMs can be taught not only to reason but also
how to reason, dynamically selecting the optimal reasoning pathway for each
task.
68c3905efc1747b912403a5c
Bohao Tang
68c3905efc1747b912403a5d
Yan Ma
claimed_verified
2025-09-12T16:07:49.574Z
633fc70529b5a95f6e15a6b7
https://cdn-avatars.huggingface.co/v1/production/uploads/633fc70529b5a95f6e15a6b7/Fzh7wWuqU-fBbzdupOUtF.jpeg
Yan Ma
user
ManTle
68c3905efc1747b912403a5e
Fei Zhang
68c3905efc1747b912403a5f
Jiadi Su
68c3905efc1747b912403a60
Ethan Chern
68c3905efc1747b912403a61
Zhulin Hu
68c3905efc1747b912403a62
Zhixin Wang
68c3905efc1747b912403a63
Pengfei Liu
68c3905efc1747b912403a64
Ya Zhang
Visual Programmability: A Guide for Code-as-Thought in Chart
  Understanding
Chart understanding presents a critical test to the reasoning capabilities of
Vision-Language Models (VLMs). Prior approaches face critical limitations: some
rely on external tools, making them brittle and constrained by a predefined
toolkit, while others fine-tune specialist models that often adopt a single
reasoning strategy, such as text-based chain-of-thought (CoT). The intermediate
steps of text-based reasoning are difficult to verify, which complicates the
use of reinforcement-learning signals that reward factual accuracy. To address
this, we propose a Code-as-Thought (CaT) approach to represent the visual
information of a chart in a verifiable, symbolic format. Our key insight is
that this strategy must be adaptive: a fixed, code-only implementation
consistently fails on complex charts where symbolic representation is
unsuitable. This finding leads us to introduce Visual Programmability: a
learnable property that determines if a chart-question pair is better solved
with code or direct visual analysis. We implement this concept in an adaptive
framework where a VLM learns to choose between the CaT pathway and a direct
visual reasoning pathway. The selection policy of the model is trained with
reinforcement learning using a novel dual-reward system. This system combines a
data-accuracy reward to ground the model in facts and prevent numerical
hallucination, with a decision reward that teaches the model when to use each
strategy, preventing it from defaulting to a single reasoning mode. Experiments
demonstrate strong and robust performance across diverse chart-understanding
benchmarks. Our work shows that VLMs can be taught not only to reason but also
how to reason, dynamically selecting the optimal reasoning pathway for each
task.
VLMs are enhanced with an adaptive framework that selects between code-based and direct visual reasoning for chart understanding, improving performance and robustness.
Code-as-Thought Visual Programmability direct visual reasoning reinforcement learning dual-reward system numerical hallucination chart-understanding benchmarks