L4GM: Large 4D Gaussian Reconstruction Model
We present L4GM, the first 4D Large Reconstruction Model that produces
animated objects from a single-view video input -- in a single feed-forward
pass that takes only a second. Key to our success is a novel dataset of
multiview videos containing curated, rendered animated objects from Objaverse.
This dataset depicts 44K diverse objects with 110K animations rendered in 48
viewpoints, resulting in 12M videos with a total of 300M frames. We keep our
L4GM simple for scalability and build directly on top of LGM, a pretrained 3D
Large Reconstruction Model that outputs 3D Gaussian ellipsoids from multiview
image input. L4GM outputs a per-frame 3D Gaussian Splatting representation from
video frames sampled at a low fps and then upsamples the representation to a
higher fps to achieve temporal smoothness. We add temporal self-attention
layers to the base LGM to help it learn consistency across time, and utilize a
per-timestep multiview rendering loss to train the model. The representation is
upsampled to a higher framerate by training an interpolation model which
produces intermediate 3D Gaussian representations. We showcase that L4GM that
is only trained on synthetic data generalizes extremely well on in-the-wild
videos, producing high quality animated 3D assets.
6670fa1cc718469ee73cac89
Jiawei Ren
admin_assigned
2024-06-18T21:36:20.287Z
64bf93139a69e8da48146cce
/avatars/0be0719e77312b12b4b3520f98ed808b.svg
Jiawei Ren
user
Jiawei1992
6670fa1cc718469ee73cac8a
Kevin Xie
6670fa1cc718469ee73cac8b
Ashkan Mirzaei
admin_assigned
2024-06-18T21:36:37.220Z
634db15dd00bb5d92c3bd94f
/avatars/9a6a1231bc5205911272d83527593f1a.svg
Ashkan Mirzaei
user
ashmrz
6670fa1cc718469ee73cac8c
Hanxue Liang
admin_assigned
2024-06-18T21:36:46.070Z
64499e6e6fa5e1ba98a309e2
/avatars/17ac2b15e42ea927de775d183d57359d.svg
liang
user
hanxue
6670fa1cc718469ee73cac8d
Xiaohui Zeng
claimed_verified
2024-11-16T19:47:22.615Z
63363cb8a8048332fdc268cd
/avatars/a9bf6ce3394a8b76eade1a7517b37fa1.svg
xiaohui zeng
user
xiaohui2022
6670fa1cc718469ee73cac8e
Karsten Kreis
6670fa1cc718469ee73cac8f
Ziwei Liu
6670fa1cc718469ee73cac90
Antonio Torralba
6670fa1cc718469ee73cac91
Sanja Fidler
6670fa1cc718469ee73cac92
Seung Wook Kim
6670fa1cc718469ee73cac93
Huan Ling
L4GM: Large 4D Gaussian Reconstruction Model
We present L4GM, the first 4D Large Reconstruction Model that produces
animated objects from a single-view video input -- in a single feed-forward
pass that takes only a second. Key to our success is a novel dataset of
multiview videos containing curated, rendered animated objects from Objaverse.
This dataset depicts 44K diverse objects with 110K animations rendered in 48
viewpoints, resulting in 12M videos with a total of 300M frames. We keep our
L4GM simple for scalability and build directly on top of LGM, a pretrained 3D
Large Reconstruction Model that outputs 3D Gaussian ellipsoids from multiview
image input. L4GM outputs a per-frame 3D Gaussian Splatting representation from
video frames sampled at a low fps and then upsamples the representation to a
higher fps to achieve temporal smoothness. We add temporal self-attention
layers to the base LGM to help it learn consistency across time, and utilize a
per-timestep multiview rendering loss to train the model. The representation is
upsampled to a higher framerate by training an interpolation model which
produces intermediate 3D Gaussian representations. We showcase that L4GM that
is only trained on synthetic data generalizes extremely well on in-the-wild
videos, producing high quality animated 3D assets.
L4GM, a 4D large reconstruction model, generates animated 3D objects from single-view video in a single feed-forward pass using 3D Gaussian splatting and temporal self-attention, demonstrating strong generalization on real-world videos.
L4GM 4D Large Reconstruction Model Gaussian splatting temporal self-attention multiview rendering loss interpolation model 3D Gaussian ellipsoids