VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D
  Reconstruction
The rapid advancement of Large Multimodal Models (LMMs) for 2D images and
videos has motivated extending these models to understand 3D scenes, aiming for
human-like visual-spatial intelligence. Nevertheless, achieving deep spatial
understanding comparable to human capabilities poses significant challenges in
model encoding and data acquisition. Existing methods frequently depend on
external depth sensors for geometry capture or utilize off-the-shelf algorithms
for pre-constructing 3D maps, thereby limiting their scalability, especially
with prevalent monocular video inputs and for time-sensitive applications. In
this work, we introduce VLM-3R, a unified framework for Vision-Language Models
(VLMs) that incorporates 3D Reconstructive instruction tuning. VLM-3R processes
monocular video frames by employing a geometry encoder to derive implicit 3D
tokens that represent spatial understanding. Leveraging our Spatial-Visual-View
Fusion and over 200K curated 3D reconstructive instruction tuning
question-answer (QA) pairs, VLM-3R effectively aligns real-world spatial
context with language instructions. This enables monocular 3D spatial
assistance and embodied reasoning. To facilitate the evaluation of temporal
reasoning, we introduce the Vision-Spatial-Temporal Intelligence benchmark,
featuring over 138.6K QA pairs across five distinct tasks focused on evolving
spatial relationships. Extensive experiments demonstrate that our model,
VLM-3R, not only facilitates robust visual-spatial reasoning but also enables
the understanding of temporal 3D context changes, excelling in both accuracy
and scalability.
6837611a3216d45c6288350c
Zhiwen Fan
6837611a3216d45c6288350d
Jian Zhang
6837611a3216d45c6288350e
Renjie Li
claimed_verified
2025-07-11T08:09:12.268Z
686fe2365998cd8af12fc8ba
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/GQbxskSw-98mdjbM3szaC.png
Renjie Li
user
renjie-li
6837611a3216d45c6288350f
Junge Zhang
6837611a3216d45c62883510
Runjin Chen
6837611a3216d45c62883511
Hezhen Hu
6837611a3216d45c62883512
Kevin Wang
6837611a3216d45c62883513
Huaizhi Qu
claimed_verified
2025-06-02T07:48:30.840Z
6621ea7224210791d94ebde4
/avatars/6510b1faa5042755ba80cf3c54b08781.svg
Huaizhi Qu
user
qhz991029
6837611a3216d45c62883514
Dilin Wang
6837611a3216d45c62883515
Zhicheng Yan
6837611a3216d45c62883516
Hongyu Xu
6837611a3216d45c62883517
Justin Theiss
6837611a3216d45c62883518
Tianlong Chen
6837611a3216d45c62883519
Jiachen Li
6837611a3216d45c6288351a
Zhengzhong Tu
claimed_verified
2025-06-03T08:50:56.517Z
62548d5fef3debb2ddf91217
/avatars/14975b45568f9c399c92c3986b6ce83e.svg
Zhengzhong Tu
user
vztu
6837611a3216d45c6288351b
Zhangyang Wang
6837611a3216d45c6288351c
Rakesh Ranjan
VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D
  Reconstruction
The rapid advancement of Large Multimodal Models (LMMs) for 2D images and
videos has motivated extending these models to understand 3D scenes, aiming for
human-like visual-spatial intelligence. Nevertheless, achieving deep spatial
understanding comparable to human capabilities poses significant challenges in
model encoding and data acquisition. Existing methods frequently depend on
external depth sensors for geometry capture or utilize off-the-shelf algorithms
for pre-constructing 3D maps, thereby limiting their scalability, especially
with prevalent monocular video inputs and for time-sensitive applications. In
this work, we introduce VLM-3R, a unified framework for Vision-Language Models
(VLMs) that incorporates 3D Reconstructive instruction tuning. VLM-3R processes
monocular video frames by employing a geometry encoder to derive implicit 3D
tokens that represent spatial understanding. Leveraging our Spatial-Visual-View
Fusion and over 200K curated 3D reconstructive instruction tuning
question-answer (QA) pairs, VLM-3R effectively aligns real-world spatial
context with language instructions. This enables monocular 3D spatial
assistance and embodied reasoning. To facilitate the evaluation of temporal
reasoning, we introduce the Vision-Spatial-Temporal Intelligence benchmark,
featuring over 138.6K QA pairs across five distinct tasks focused on evolving
spatial relationships. Extensive experiments demonstrate that our model,
VLM-3R, not only facilitates robust visual-spatial reasoning but also enables
the understanding of temporal 3D context changes, excelling in both accuracy
and scalability.
VLM-3R, a framework for Vision-Language Models, incorporates 3D reconstructive instruction tuning to process monocular video frames and perform embodied reasoning with robust visual-spatial and temporal contextual understanding.
Large Multimodal Models Vision-Language Models 3D Reconstructive instruction tuning geometry encoder implicit 3D tokens Spatial-Visual-View Fusion Vision-Spatial-Temporal Intelligence benchmark embodied reasoning