Improved Iterative Refinement for Chart-to-Code Generation via
  Structured Instruction
Recently, multimodal large language models (MLLMs) have attracted increasing
research attention due to their powerful visual understanding capabilities.
While they have achieved impressive results on various vision tasks, their
performance on chart-to-code generation remains suboptimal. This task requires
MLLMs to generate executable code that can reproduce a given chart, demanding
not only precise visual understanding but also accurate translation of visual
elements into structured code. Directly prompting MLLMs to perform this complex
task often yields unsatisfactory results. To address this challenge, we propose
{ChartIR}, an iterative refinement method based on structured instruction.
First, we distinguish two tasks: visual understanding and code translation. To
accomplish the visual understanding component, we design two types of
structured instructions: description and difference. The description
instruction captures the visual elements of the reference chart, while the
difference instruction characterizes the discrepancies between the reference
chart and the generated chart. These instructions effectively transform visual
features into language representations, thereby facilitating the subsequent
code translation process. Second, we decompose the overall chart generation
pipeline into two stages: initial code generation and iterative refinement,
enabling progressive enhancement of the final output. Experimental results show
that, compared to other method, our method achieves superior performance on
both the open-source model Qwen2-VL and the closed-source model GPT-4o.
6854ea7a7bc8d012d4ca998d
Chengzhi Xu
admin_assigned
2025-06-20T12:30:45.306Z
672a29efe53061b3dc76fd70
/avatars/20c7100884f4a69a9ec781315f68ff0b.svg
xuchengzhi
user
dazhiga
6854ea7a7bc8d012d4ca998e
Yuyang Wang
6854ea7a7bc8d012d4ca998f
Lai Wei
claimed_verified
2025-06-20T08:57:40.628Z
64a16b1aeacb4b50ba1c889d
https://cdn-avatars.huggingface.co/v1/production/uploads/64a16b1aeacb4b50ba1c889d/EhWoBGL6LlFGIVTUsp2F4.jpeg
Lai Wei
user
WaltonFuture
6854ea7a7bc8d012d4ca9990
Lichao Sun
admin_assigned
2025-06-20T12:30:53.972Z
65a52766215aabac489e3468
/avatars/fe05e22cd7e12e961296426434e17c76.svg
Lichao Sun
user
sunlichao137
6854ea7a7bc8d012d4ca9991
Weiran Huang
admin_assigned
2025-06-20T12:31:00.023Z
65e095da35ad8b2fe8c80e71
/avatars/225d4be6411dcec2460047ea1a88a5c3.svg
Weiran Huang
user
weiranhuang
Improved Iterative Refinement for Chart-to-Code Generation via
  Structured Instruction
Recently, multimodal large language models (MLLMs) have attracted increasing
research attention due to their powerful visual understanding capabilities.
While they have achieved impressive results on various vision tasks, their
performance on chart-to-code generation remains suboptimal. This task requires
MLLMs to generate executable code that can reproduce a given chart, demanding
not only precise visual understanding but also accurate translation of visual
elements into structured code. Directly prompting MLLMs to perform this complex
task often yields unsatisfactory results. To address this challenge, we propose
{ChartIR}, an iterative refinement method based on structured instruction.
First, we distinguish two tasks: visual understanding and code translation. To
accomplish the visual understanding component, we design two types of
structured instructions: description and difference. The description
instruction captures the visual elements of the reference chart, while the
difference instruction characterizes the discrepancies between the reference
chart and the generated chart. These instructions effectively transform visual
features into language representations, thereby facilitating the subsequent
code translation process. Second, we decompose the overall chart generation
pipeline into two stages: initial code generation and iterative refinement,
enabling progressive enhancement of the final output. Experimental results show
that, compared to other method, our method achieves superior performance on
both the open-source model Qwen2-VL and the closed-source model GPT-4o.
ChartIR uses structured instruction and iterative refinement to improve MLLM performance in chart-to-code generation by separating visual understanding and code translation tasks.
multimodal large language models MLLMs visual understanding code translation structured instruction description instruction difference instruction language representations initial code generation iterative refinement Qwen2-VL GPT-4o