Thinking with Generated Images
We present Thinking with Generated Images, a novel paradigm that
fundamentally transforms how large multimodal models (LMMs) engage with visual
reasoning by enabling them to natively think across text and vision modalities
through spontaneous generation of intermediate visual thinking steps. Current
visual reasoning with LMMs is constrained to either processing fixed
user-provided images or reasoning solely through text-based chain-of-thought
(CoT). Thinking with Generated Images unlocks a new dimension of cognitive
capability where models can actively construct intermediate visual thoughts,
critique their own visual hypotheses, and refine them as integral components of
their reasoning process. We demonstrate the effectiveness of our approach
through two complementary mechanisms: (1) vision generation with intermediate
visual subgoals, where models decompose complex visual tasks into manageable
components that are generated and integrated progressively, and (2) vision
generation with self-critique, where models generate an initial visual
hypothesis, analyze its shortcomings through textual reasoning, and produce
refined outputs based on their own critiques. Our experiments on vision
generation benchmarks show substantial improvements over baseline approaches,
with our models achieving up to 50% (from 38% to 57%) relative improvement in
handling complex multi-object scenarios. From biochemists exploring novel
protein structures, and architects iterating on spatial designs, to forensic
analysts reconstructing crime scenes, and basketball players envisioning
strategic plays, our approach enables AI models to engage in the kind of visual
imagination and iterative refinement that characterizes human creative,
analytical, and strategic thinking. We release our open-source suite at
https://github.com/GAIR-NLP/thinking-with-generated-images.
6837da438f680552f7b86b28
Ethan Chern
claimed_verified
2025-05-29T07:42:05.613Z
64bb5f9d8e051085bace4d1e
/avatars/15ccbb78c6131dfe46b7a9d8e7d1a31f.svg
Ethan Chern
user
ethanchern
6837da438f680552f7b86b29
Zhulin Hu
6837da438f680552f7b86b2a
Steffi Chern
admin_assigned
2025-05-30T07:03:14.750Z
64b370fe6d953e7c75ede314
https://cdn-avatars.huggingface.co/v1/production/uploads/64b370fe6d953e7c75ede314/RdP2q3hGXWE4E2zfSv0KU.png
Steffi Chern
user
steffichern
6837da438f680552f7b86b2b
Siqi Kou
admin_assigned
2025-05-30T07:03:25.399Z
654e330f350abceb30a1390b
/avatars/e54a8be788fa1bdc7acefecc208215bb.svg
KouSiqi
user
karrykkk
6837da438f680552f7b86b2c
Jiadi Su
6837da438f680552f7b86b2d
Yan Ma
claimed_verified
2025-05-30T06:54:53.066Z
633fc70529b5a95f6e15a6b7
https://cdn-avatars.huggingface.co/v1/production/uploads/633fc70529b5a95f6e15a6b7/Fzh7wWuqU-fBbzdupOUtF.jpeg
Yan Ma
user
ManTle
6837da438f680552f7b86b2e
Zhijie Deng
admin_assigned
2025-05-30T07:03:34.787Z
673d5f411b0fe168ad4896b2
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/fYgQx6XNi_P5GxlUKbH5G.png
Zhijie Deng
user
thudzj
6837da438f680552f7b86b2f
Pengfei Liu
admin_assigned
2025-05-30T07:03:48.958Z
6144a0c4ff1146bbd84d9865
https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
Pengfei Liu
user
Pengfei
Thinking with Generated Images
We present Thinking with Generated Images, a novel paradigm that
fundamentally transforms how large multimodal models (LMMs) engage with visual
reasoning by enabling them to natively think across text and vision modalities
through spontaneous generation of intermediate visual thinking steps. Current
visual reasoning with LMMs is constrained to either processing fixed
user-provided images or reasoning solely through text-based chain-of-thought
(CoT). Thinking with Generated Images unlocks a new dimension of cognitive
capability where models can actively construct intermediate visual thoughts,
critique their own visual hypotheses, and refine them as integral components of
their reasoning process. We demonstrate the effectiveness of our approach
through two complementary mechanisms: (1) vision generation with intermediate
visual subgoals, where models decompose complex visual tasks into manageable
components that are generated and integrated progressively, and (2) vision
generation with self-critique, where models generate an initial visual
hypothesis, analyze its shortcomings through textual reasoning, and produce
refined outputs based on their own critiques. Our experiments on vision
generation benchmarks show substantial improvements over baseline approaches,
with our models achieving up to 50% (from 38% to 57%) relative improvement in
handling complex multi-object scenarios. From biochemists exploring novel
protein structures, and architects iterating on spatial designs, to forensic
analysts reconstructing crime scenes, and basketball players envisioning
strategic plays, our approach enables AI models to engage in the kind of visual
imagination and iterative refinement that characterizes human creative,
analytical, and strategic thinking. We release our open-source suite at
https://github.com/GAIR-NLP/thinking-with-generated-images.
Thinking with Generated Images allows large multimodal models to generate and critique intermediate visual steps, enhancing visual reasoning capabilities and achieving significant improvements in complex scenarios.
LMMs visual reasoning chain-of-thought vision generation intermediate visual subgoals self-critique multis-object scenarios biochemists architects forensic analysts basketball players