PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on
  Structured Images
Structured images (e.g., charts and geometric diagrams) remain challenging
for multimodal large language models (MLLMs), as perceptual slips can cascade
into erroneous conclusions. Intermediate visual cues can steer reasoning;
however, existing cue-based methods are constrained with low-fidelity image
processing and linear, rigid reasoning patterns, limiting their effectiveness
on complex structured-image tasks. In this paper, we propose PixelCraft, a
novel multi-agent system for high-fidelity image processing and flexible visual
reasoning on structured images. The system comprises a dispatcher, a planner, a
reasoner, critics, and a set of visual tool agents. To achieve high-fidelity
processing, we construct a high-quality corpus and fine-tune an MLLM into a
grounding model, whose pixel-level localizations are integrated with
traditional computer vision (CV) algorithms in tool agents. Building on this
foundation, PixelCraft facilitates flexible visual reasoning through a dynamic
three-stage workflow of tool selection, agent discussion, and self-criticism.
Moreover, unlike prior linear reasoning patterns that simply append historical
images, PixelCraft maintains an image memory to allow the planner to adaptively
revisit earlier visual steps, explore alternative reasoning branches, and
dynamically adjust the reasoning trajectory during discussion. Extensive
experiments on challenging chart and geometry benchmarks demonstrate that
PixelCraft significantly improves visual reasoning performance for advanced
MLLMs, setting a new standard for structured image reasoning. Our code will be
available at https://github.com/microsoft/PixelCraft.
68db4dacd2bf1f4b15ec7471
Shuoshuo Zhang
68db4dacd2bf1f4b15ec7472
Zijian Li
claimed_verified
2025-09-30T06:27:54.665Z
64d31c55d8b712baf198602f
/avatars/fad7972744f879116a8dc8b406f8b91c.svg
Zijian Li
user
zli999
68db4dacd2bf1f4b15ec7473
Yizhen Zhang
68db4dacd2bf1f4b15ec7474
Jingjing Fu
68db4dacd2bf1f4b15ec7475
Lei Song
68db4dacd2bf1f4b15ec7476
Jiang Bian
68db4dacd2bf1f4b15ec7477
Jun Zhang
68db4dacd2bf1f4b15ec7478
Yujiu Yang
68db4dacd2bf1f4b15ec7479
Rui Wang
PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on
  Structured Images
Structured images (e.g., charts and geometric diagrams) remain challenging
for multimodal large language models (MLLMs), as perceptual slips can cascade
into erroneous conclusions. Intermediate visual cues can steer reasoning;
however, existing cue-based methods are constrained with low-fidelity image
processing and linear, rigid reasoning patterns, limiting their effectiveness
on complex structured-image tasks. In this paper, we propose PixelCraft, a
novel multi-agent system for high-fidelity image processing and flexible visual
reasoning on structured images. The system comprises a dispatcher, a planner, a
reasoner, critics, and a set of visual tool agents. To achieve high-fidelity
processing, we construct a high-quality corpus and fine-tune an MLLM into a
grounding model, whose pixel-level localizations are integrated with
traditional computer vision (CV) algorithms in tool agents. Building on this
foundation, PixelCraft facilitates flexible visual reasoning through a dynamic
three-stage workflow of tool selection, agent discussion, and self-criticism.
Moreover, unlike prior linear reasoning patterns that simply append historical
images, PixelCraft maintains an image memory to allow the planner to adaptively
revisit earlier visual steps, explore alternative reasoning branches, and
dynamically adjust the reasoning trajectory during discussion. Extensive
experiments on challenging chart and geometry benchmarks demonstrate that
PixelCraft significantly improves visual reasoning performance for advanced
MLLMs, setting a new standard for structured image reasoning. Our code will be
available at https://github.com/microsoft/PixelCraft.
PixelCraft, a multi-agent system, enhances visual reasoning in multimodal large language models by integrating high-fidelity image processing and flexible reasoning through a dynamic workflow and image memory.
multimodal large language models perceptual slips intermediate visual cues multi-agent system high-fidelity image processing flexible visual reasoning dispatcher planner reasoner critics visual tool agents grounding model pixel-level localizations traditional computer vision dynamic three-stage workflow tool selection agent discussion self-criticism image memory visual reasoning performance structured image reasoning