SViM3D: Stable Video Material Diffusion for Single Image 3D Generation
We present Stable Video Materials 3D (SViM3D), a framework to predict
multi-view consistent physically based rendering (PBR) materials, given a
single image. Recently, video diffusion models have been successfully used to
reconstruct 3D objects from a single image efficiently. However, reflectance is
still represented by simple material models or needs to be estimated in
additional steps to enable relighting and controlled appearance edits. We
extend a latent video diffusion model to output spatially varying PBR
parameters and surface normals jointly with each generated view based on
explicit camera control. This unique setup allows for relighting and generating
a 3D asset using our model as neural prior. We introduce various mechanisms to
this pipeline that improve quality in this ill-posed setting. We show
state-of-the-art relighting and novel view synthesis performance on multiple
object-centric datasets. Our method generalizes to diverse inputs, enabling the
generation of relightable 3D assets useful in AR/VR, movies, games and other
visual media.
68e8dd4d95e8e6771df38c30
Andreas Engelhardt
68e8dd4d95e8e6771df38c31
Mark Boss
68e8dd4d95e8e6771df38c32
Vikram Voletti
68e8dd4d95e8e6771df38c33
Chun-Han Yao
68e8dd4d95e8e6771df38c34
Hendrik P. A. Lensch
68e8dd4d95e8e6771df38c35
Varun Jampani
SViM3D: Stable Video Material Diffusion for Single Image 3D Generation
We present Stable Video Materials 3D (SViM3D), a framework to predict
multi-view consistent physically based rendering (PBR) materials, given a
single image. Recently, video diffusion models have been successfully used to
reconstruct 3D objects from a single image efficiently. However, reflectance is
still represented by simple material models or needs to be estimated in
additional steps to enable relighting and controlled appearance edits. We
extend a latent video diffusion model to output spatially varying PBR
parameters and surface normals jointly with each generated view based on
explicit camera control. This unique setup allows for relighting and generating
a 3D asset using our model as neural prior. We introduce various mechanisms to
this pipeline that improve quality in this ill-posed setting. We show
state-of-the-art relighting and novel view synthesis performance on multiple
object-centric datasets. Our method generalizes to diverse inputs, enabling the
generation of relightable 3D assets useful in AR/VR, movies, games and other
visual media.
A latent video diffusion model predicts multi-view consistent PBR materials from a single image, enabling relighting and novel view synthesis with high quality.
video diffusion models physically based rendering (PBR) spatially varying PBR parameters surface normals explicit camera control neural prior relighting novel view synthesis object-centric datasets AR/VR movies games visual media