Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery
  Simulation
We present a framework for perspective-aware reasoning in vision-language
models (VLMs) through mental imagery simulation. Perspective-taking, the
ability to perceive an environment or situation from an alternative viewpoint,
is a key benchmark for human-level visual understanding, essential for
environmental interaction and collaboration with autonomous agents. Despite
advancements in spatial reasoning within VLMs, recent research has shown that
modern VLMs significantly lack perspective-aware reasoning capabilities and
exhibit a strong bias toward egocentric interpretations. To bridge the gap
between VLMs and human perception, we focus on the role of mental imagery,
where humans perceive the world through abstracted representations that
facilitate perspective shifts. Motivated by this, we propose a framework for
perspective-aware reasoning, named Abstract Perspective Change (APC), that
effectively leverages vision foundation models, such as object detection,
segmentation, and orientation estimation, to construct scene abstractions and
enable perspective transformations. Our experiments on synthetic and real-image
benchmarks, compared with various VLMs, demonstrate significant improvements in
perspective-aware reasoning with our framework, further outperforming
fine-tuned spatial reasoning models and novel-view-synthesis-based approaches.
680af2bf2fa10fbf21684bde
Phillip Y. Lee
claimed_verified
2025-04-26T08:53:40.936Z
6342796a0875f2c99cfd313b
/avatars/98575092404c4197b20c929a6499a015.svg
Yuseung "Phillip" Lee
user
phillipinseoul
680af2bf2fa10fbf21684bdf
Jihyeon Je
680af2bf2fa10fbf21684be0
Chanho Park
680af2bf2fa10fbf21684be1
Mikaela Angelina Uy
680af2bf2fa10fbf21684be2
Leonidas Guibas
680af2bf2fa10fbf21684be3
Minhyuk Sung
Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery
  Simulation
We present a framework for perspective-aware reasoning in vision-language
models (VLMs) through mental imagery simulation. Perspective-taking, the
ability to perceive an environment or situation from an alternative viewpoint,
is a key benchmark for human-level visual understanding, essential for
environmental interaction and collaboration with autonomous agents. Despite
advancements in spatial reasoning within VLMs, recent research has shown that
modern VLMs significantly lack perspective-aware reasoning capabilities and
exhibit a strong bias toward egocentric interpretations. To bridge the gap
between VLMs and human perception, we focus on the role of mental imagery,
where humans perceive the world through abstracted representations that
facilitate perspective shifts. Motivated by this, we propose a framework for
perspective-aware reasoning, named Abstract Perspective Change (APC), that
effectively leverages vision foundation models, such as object detection,
segmentation, and orientation estimation, to construct scene abstractions and
enable perspective transformations. Our experiments on synthetic and real-image
benchmarks, compared with various VLMs, demonstrate significant improvements in
perspective-aware reasoning with our framework, further outperforming
fine-tuned spatial reasoning models and novel-view-synthesis-based approaches.
A framework named Abstract Perspective Change (APC) enhances perspective-aware reasoning in vision-language models by utilizing vision foundation models to create scene abstractions and enable perspective shifts.
vision-language models mental imagery simulation perspective-taking spatial reasoning scene abstractions perspective transformations object detection segmentation orientation estimation synthesis Abstract Perspective Change (APC)