Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model
  Self-Distillation
The ability to generate virtual environments is crucial for applications
ranging from gaming to physical AI domains such as robotics, autonomous
driving, and industrial AI. Current learning-based 3D reconstruction methods
rely on the availability of captured real-world multi-view data, which is not
always readily available. Recent advancements in video diffusion models have
shown remarkable imagination capabilities, yet their 2D nature limits the
applications to simulation where a robot needs to navigate and interact with
the environment. In this paper, we propose a self-distillation framework that
aims to distill the implicit 3D knowledge in the video diffusion models into an
explicit 3D Gaussian Splatting (3DGS) representation, eliminating the need for
multi-view training data. Specifically, we augment the typical RGB decoder with
a 3DGS decoder, which is supervised by the output of the RGB decoder. In this
approach, the 3DGS decoder can be purely trained with synthetic data generated
by video diffusion models. At inference time, our model can synthesize 3D
scenes from either a text prompt or a single image for real-time rendering. Our
framework further extends to dynamic 3D scene generation from a monocular input
video. Experimental results show that our framework achieves state-of-the-art
performance in static and dynamic 3D scene generation.
68d34f1a0e215259d193b19c
Sherwin Bahmani
admin_assigned
2025-09-24T15:27:57.343Z
647ef66c45baf21ad707b291
/avatars/4cd941cdca6dd829fdc9cb3fb788a99c.svg
Sherwin Bahmani
user
sherwinbahmani
68d34f1a0e215259d193b19d
Tianchang Shen
68d34f1a0e215259d193b19e
Jiawei Ren
68d34f1a0e215259d193b19f
Jiahui Huang
68d34f1a0e215259d193b1a0
Yifeng Jiang
68d34f1a0e215259d193b1a1
Haithem Turki
admin_assigned
2025-09-24T15:29:09.869Z
656e000253703dd78fd072a9
/avatars/6702ba8fabe3d08884aa757f90cea333.svg
Haithem Turki
user
hturki
68d34f1a0e215259d193b1a2
Andrea Tagliasacchi
68d34f1a0e215259d193b1a3
David B. Lindell
68d34f1a0e215259d193b1a4
Zan Gojcic
admin_assigned
2025-09-24T15:28:51.842Z
6366cda3361a96184dc22139
/avatars/d8a88c84cb5f69e69dd038674a29be89.svg
Zan Gojcic
user
zgojcic
68d34f1a0e215259d193b1a5
Sanja Fidler
68d34f1a0e215259d193b1a6
Huan Ling
68d34f1a0e215259d193b1a7
Jun Gao
68d34f1a0e215259d193b1a8
Xuanchi Ren
claimed_verified
2025-09-25T07:23:07.715Z
658529d61c461dfe88afe8e8
/avatars/a22c1b07d28c2662833c462c6537d835.svg
Xuanchi Ren
user
xrenaa
Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model
  Self-Distillation
The ability to generate virtual environments is crucial for applications
ranging from gaming to physical AI domains such as robotics, autonomous
driving, and industrial AI. Current learning-based 3D reconstruction methods
rely on the availability of captured real-world multi-view data, which is not
always readily available. Recent advancements in video diffusion models have
shown remarkable imagination capabilities, yet their 2D nature limits the
applications to simulation where a robot needs to navigate and interact with
the environment. In this paper, we propose a self-distillation framework that
aims to distill the implicit 3D knowledge in the video diffusion models into an
explicit 3D Gaussian Splatting (3DGS) representation, eliminating the need for
multi-view training data. Specifically, we augment the typical RGB decoder with
a 3DGS decoder, which is supervised by the output of the RGB decoder. In this
approach, the 3DGS decoder can be purely trained with synthetic data generated
by video diffusion models. At inference time, our model can synthesize 3D
scenes from either a text prompt or a single image for real-time rendering. Our
framework further extends to dynamic 3D scene generation from a monocular input
video. Experimental results show that our framework achieves state-of-the-art
performance in static and dynamic 3D scene generation.
A self-distillation framework converts implicit 3D knowledge from video diffusion models into an explicit 3D Gaussian Splatting representation, enabling 3D scene generation from text or images.
video diffusion models 3D Gaussian Splatting 3DGS RGB decoder 3D scene generation text prompt monocular input video dynamic 3D scene generation