HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Video
Digitizing the physical world into accurate simulation-ready virtual
environments offers significant opportunities in a variety of fields such as
augmented and virtual reality, gaming, and robotics. However, current 3D
reconstruction and scene-understanding methods commonly fall short in one or
more critical aspects, such as geometry completeness, object interactivity,
physical plausibility, photorealistic rendering, or realistic physical
properties for reliable dynamic simulation. To address these limitations, we
introduce HoloScene, a novel interactive 3D reconstruction framework that
simultaneously achieves these requirements. HoloScene leverages a comprehensive
interactive scene-graph representation, encoding object geometry, appearance,
and physical properties alongside hierarchical and inter-object relationships.
Reconstruction is formulated as an energy-based optimization problem,
integrating observational data, physical constraints, and generative priors
into a unified, coherent objective. Optimization is efficiently performed via a
hybrid approach combining sampling-based exploration with gradient-based
refinement. The resulting digital twins exhibit complete and precise geometry,
physical stability, and realistic rendering from novel viewpoints. Evaluations
conducted on multiple benchmark datasets demonstrate superior performance,
while practical use-cases in interactive gaming and real-time digital-twin
manipulation illustrate HoloScene's broad applicability and effectiveness.
Project page: https://xiahongchi.github.io/HoloScene.
68e5ca4e975ac4c405ef20e9
Hongchi Xia
68e5ca4e975ac4c405ef20ea
Chih-Hao Lin
68e5ca4e975ac4c405ef20eb
Hao-Yu Hsu
68e5ca4e975ac4c405ef20ec
Quentin Leboutet
68e5ca4e975ac4c405ef20ed
Katelyn Gao
68e5ca4e975ac4c405ef20ee
Michael Paulitsch
68e5ca4e975ac4c405ef20ef
Benjamin Ummenhofer
68e5ca4e975ac4c405ef20f0
Shenlong Wang
HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Video
Digitizing the physical world into accurate simulation-ready virtual
environments offers significant opportunities in a variety of fields such as
augmented and virtual reality, gaming, and robotics. However, current 3D
reconstruction and scene-understanding methods commonly fall short in one or
more critical aspects, such as geometry completeness, object interactivity,
physical plausibility, photorealistic rendering, or realistic physical
properties for reliable dynamic simulation. To address these limitations, we
introduce HoloScene, a novel interactive 3D reconstruction framework that
simultaneously achieves these requirements. HoloScene leverages a comprehensive
interactive scene-graph representation, encoding object geometry, appearance,
and physical properties alongside hierarchical and inter-object relationships.
Reconstruction is formulated as an energy-based optimization problem,
integrating observational data, physical constraints, and generative priors
into a unified, coherent objective. Optimization is efficiently performed via a
hybrid approach combining sampling-based exploration with gradient-based
refinement. The resulting digital twins exhibit complete and precise geometry,
physical stability, and realistic rendering from novel viewpoints. Evaluations
conducted on multiple benchmark datasets demonstrate superior performance,
while practical use-cases in interactive gaming and real-time digital-twin
manipulation illustrate HoloScene's broad applicability and effectiveness.
Project page: https://xiahongchi.github.io/HoloScene.
HoloScene is an interactive 3D reconstruction framework that achieves geometry completeness, object interactivity, physical plausibility, photorealistic rendering, and realistic physical properties through an energy-based optimization problem.
interactive 3D reconstruction scene-graph representation energy-based optimization sampling-based exploration gradient-based refinement digital twins geometry completeness physical stability photorealistic rendering