Video Perception Model for 3D Scene Synthesis
Rui Huang, Guangyao Zhai, Zuria Bauer, Marc Pollefeys, Federico Tombari, Leonidas Guibas, Gao Huang, Francis Engelmann
3D scene synthesis has traditionally required expert knowledge and considerable manual effort; automating this process holds the potential to greatly advance applications in architectural design, robotics simulation, virtual reality, and gaming.Most recent 3D scene synthesis models tap into the commonsense knowledge encoded in Large Language Models (LLMs) or leverage strong appearance priors of modern image generation models.However, current LLMs exhibit limited 3D spatial reasoning ability,which hinders their ability to generate realistic and coherent 3D scenes.Meanwhile, image generation-based methods often suffer from multi-view inconsistencies.In this work, we introduce ${\textbf{Vi}}$deo ${\textbf{P}}$erception Models for 3D $\textbf{Scene}$ synthesis ($\textbf{VIPScene}$), a novel framework that uses video generation models to leverage the encoded commonsense knowledge of the 3D physical world. VIPScene accepts both text and image prompts and seamlessly integrates video generation, feedforward 3D reconstruction, and open-vocabulary perception models to semantically and geometrically analyze each object in a scene. This enables flexible scene synthesis with high realism and structural consistency.We further introduce $\textbf{F}$irst-$\textbf{P}$erson $\textbf{V}$iew $\textbf{Score}$ $\textbf{(FPVScore)}$ for consistency and reality evaluation, utilizing continuous first-person perspective to capitalize on the reasoning ability of Multimodal Large Language Models.Extensive experiments show that VIPScene outperforms existing methods on indoor scene generation, and generalizes well to diverse room configurations. The code will be released.