Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal
  Language Models
Humans draw to facilitate reasoning: we draw auxiliary lines when solving
geometry problems; we mark and circle when reasoning on maps; we use sketches
to amplify our ideas and relieve our limited-capacity working memory. However,
such actions are missing in current multimodal language models (LMs). Current
chain-of-thought and tool-use paradigms only use text as intermediate reasoning
steps. In this work, we introduce Sketchpad, a framework that gives multimodal
LMs a visual sketchpad and tools to draw on the sketchpad. The LM conducts
planning and reasoning according to the visual artifacts it has drawn.
Different from prior work, which uses text-to-image models to enable LMs to
draw, Sketchpad enables LMs to draw with lines, boxes, marks, etc., which is
closer to human sketching and better facilitates reasoning. Sketchpad can also
use specialist vision models during the sketching process (e.g., draw bounding
boxes with object detection models, draw masks with segmentation models), to
further enhance visual perception and reasoning. We experiment with a wide
range of math tasks (including geometry, functions, graphs, and chess) and
complex visual reasoning tasks. Sketchpad substantially improves performance on
all tasks over strong base models with no sketching, yielding an average gain
of 12.7% on math tasks, and 8.6% on vision tasks. GPT-4o with Sketchpad sets a
new state of the art on all tasks, including V*Bench (80.3%), BLINK spatial
reasoning (83.9%), and visual correspondence (80.8%). All codes and data are in
https://visualsketchpad.github.io/.
666bb62ed0e6a329a3e46a1d
Yushi Hu
extracted_confirmed
2025-05-27T19:21:18.254Z
62b1474bdcbad6848a91a54e
/avatars/d7308899b46232cad4a48a0e876449a8.svg
Yushi Hu
user
yushihu
666bb62ed0e6a329a3e46a1e
Weijia Shi
admin_assigned
2024-06-14T10:06:58.721Z
6400f2ed568dbe30c9161e47
/avatars/c55938df5bce82b5d96e592a1ec36a8b.svg
Weijia Shi
user
swj0419
666bb62ed0e6a329a3e46a1f
Xingyu Fu
admin_assigned
2024-06-14T10:07:13.871Z
6336091b2db86a181ccd6054
/avatars/829f69436225d05d2c2136bc90f640d7.svg
Xingyu Fu
user
Fiaa
666bb62ed0e6a329a3e46a20
Dan Roth
admin_assigned
2024-06-14T10:07:21.248Z
63191aa1f18d0b5d999a47fc
/avatars/d7eadb22aa3d8a067189b924767864e9.svg
Dan Roth
user
danielpaulroth
666bb62ed0e6a329a3e46a21
Mari Ostendorf
666bb62ed0e6a329a3e46a22
Luke Zettlemoyer
666bb62ed0e6a329a3e46a23
Noah A Smith
666bb62ed0e6a329a3e46a24
Ranjay Krishna
admin_assigned
2024-06-14T10:07:42.545Z
66429868ab89e3a3a85668b0
/avatars/170e0daa454838deee2bf946f7118651.svg
Ranjay Krishna
user
ranjaykrishna
Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal
  Language Models
Humans draw to facilitate reasoning: we draw auxiliary lines when solving
geometry problems; we mark and circle when reasoning on maps; we use sketches
to amplify our ideas and relieve our limited-capacity working memory. However,
such actions are missing in current multimodal language models (LMs). Current
chain-of-thought and tool-use paradigms only use text as intermediate reasoning
steps. In this work, we introduce Sketchpad, a framework that gives multimodal
LMs a visual sketchpad and tools to draw on the sketchpad. The LM conducts
planning and reasoning according to the visual artifacts it has drawn.
Different from prior work, which uses text-to-image models to enable LMs to
draw, Sketchpad enables LMs to draw with lines, boxes, marks, etc., which is
closer to human sketching and better facilitates reasoning. Sketchpad can also
use specialist vision models during the sketching process (e.g., draw bounding
boxes with object detection models, draw masks with segmentation models), to
further enhance visual perception and reasoning. We experiment with a wide
range of math tasks (including geometry, functions, graphs, and chess) and
complex visual reasoning tasks. Sketchpad substantially improves performance on
all tasks over strong base models with no sketching, yielding an average gain
of 12.7% on math tasks, and 8.6% on vision tasks. GPT-4o with Sketchpad sets a
new state of the art on all tasks, including V*Bench (80.3%), BLINK spatial
reasoning (83.9%), and visual correspondence (80.8%). All codes and data are in
https://visualsketchpad.github.io/.
Sketchpad introduced a visual sketchpad with drawing tools for multimodal LMs, enhancing performance across various math and visual reasoning tasks through human-like sketching and the use of specialist vision models.
Sketchpad multimodal LMs visual sketchpad drawing tools human-like sketching specialist vision models bounding boxes segmentation models geometry functions graphs chess vision tasks V*Bench BLINK spatial reasoning visual correspondence