Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors
Duo Zheng, shijia Huang, Yanyang Li, Liwei Wang
Previous research has explored the use of Multimodal Large Language Models (MLLMs) for 3D scene understanding by treating scenes as videos. These methods typically rely on explicit dense 3D inputs, such as point clouds or reconstructed Bird's-Eye View (BEV) maps. In this work, we take an innovative step forward by questioning whether we can improve MLLMs' 3D spatial understanding and reasoning capability by directly learning from videos with 3D vision geometry priors. We introduce a simple yet effective approach: the Video-3D Geometry LLM (VG LLM). Our method utilizes a 3D visual geometry encoder to extract 3D geometry prior information from the input video sequences. This information is integrated with visual tokens and fed into the MLLM. Our approach significantly boosts the 3D perception and reasoning capabilities of MLLMs, demonstrating notable improvements in various 3D scene understanding tasks and spatial reasoning benchmarks, all learned directly from videos. Remarkably, without relying on any explicit 3D data inputs, our 4B model achieves competitive performance with prior state-of-the-art methods, even outperforming the Gemini-1.5-Pro on the VSI-Bench.