Generic 3D Diffusion Adapter Using Controlled Multi-View Editing
Open-domain 3D object synthesis has been lagging behind image synthesis due
to limited data and higher computational complexity. To bridge this gap, recent
works have investigated multi-view diffusion but often fall short in either 3D
consistency, visual quality, or efficiency. This paper proposes MVEdit, which
functions as a 3D counterpart of SDEdit, employing ancestral sampling to
jointly denoise multi-view images and output high-quality textured meshes.
Built on off-the-shelf 2D diffusion models, MVEdit achieves 3D consistency
through a training-free 3D Adapter, which lifts the 2D views of the last
timestep into a coherent 3D representation, then conditions the 2D views of the
next timestep using rendered views, without uncompromising visual quality. With
an inference time of only 2-5 minutes, this framework achieves better trade-off
between quality and speed than score distillation. MVEdit is highly versatile
and extendable, with a wide range of applications including text/image-to-3D
generation, 3D-to-3D editing, and high-quality texture synthesis. In
particular, evaluations demonstrate state-of-the-art performance in both
image-to-3D and text-guided texture generation tasks. Additionally, we
introduce a method for fine-tuning 2D latent diffusion models on small 3D
datasets with limited resources, enabling fast low-resolution text-to-3D
initialization.
65f90f3248699f59af9f97fe
Hansheng Chen
claimed_verified
2024-03-19T08:45:21.231Z
638067fcb334960c987fbeda
/avatars/87f1eaaf6b3a9c0d47d6f406261ccc18.svg
Hansheng Chen
user
Lakonik
65f90f3248699f59af9f97ff
Ruoxi Shi
admin_assigned
2024-03-19T09:44:19.854Z
640820798dca6cec91cb2f0a
/avatars/e011bc04facd251570247d1ae214dbde.svg
Ruoxi Shi
user
eliphatfs
65f90f3248699f59af9f9800
Yulin Liu
admin_assigned
2024-03-19T09:44:38.242Z
647169fb1e9659ae5c947705
/avatars/d19415d746af1d48ccbf5aafb84153c1.svg
Yulin Liu
user
yulinliu
65f90f3248699f59af9f9801
Bokui Shen
65f90f3248699f59af9f9802
Jiayuan Gu
admin_assigned
2024-03-19T09:44:56.373Z
638aa283cebef0d13aa2ec2e
/avatars/3b67e6a6073033864a817230e97c27ca.svg
Jiayuan Gu
user
jigu
65f90f3248699f59af9f9803
Gordon Wetzstein
65f90f3248699f59af9f9804
Hao Su
65f90f3248699f59af9f9805
Leonidas Guibas
Generic 3D Diffusion Adapter Using Controlled Multi-View Editing
Open-domain 3D object synthesis has been lagging behind image synthesis due
to limited data and higher computational complexity. To bridge this gap, recent
works have investigated multi-view diffusion but often fall short in either 3D
consistency, visual quality, or efficiency. This paper proposes MVEdit, which
functions as a 3D counterpart of SDEdit, employing ancestral sampling to
jointly denoise multi-view images and output high-quality textured meshes.
Built on off-the-shelf 2D diffusion models, MVEdit achieves 3D consistency
through a training-free 3D Adapter, which lifts the 2D views of the last
timestep into a coherent 3D representation, then conditions the 2D views of the
next timestep using rendered views, without uncompromising visual quality. With
an inference time of only 2-5 minutes, this framework achieves better trade-off
between quality and speed than score distillation. MVEdit is highly versatile
and extendable, with a wide range of applications including text/image-to-3D
generation, 3D-to-3D editing, and high-quality texture synthesis. In
particular, evaluations demonstrate state-of-the-art performance in both
image-to-3D and text-guided texture generation tasks. Additionally, we
introduce a method for fine-tuning 2D latent diffusion models on small 3D
datasets with limited resources, enabling fast low-resolution text-to-3D
initialization.
MVEdit uses ancestral sampling to denoise multi-view images and generate 3D textured meshes, achieving high-quality 3D consistency through a 3D Adapter and surpassing score distillation in both speed and quality.
SDEdit ancestral sampling 3D Adapter 2D diffusion models 3D consistency visual quality inference time score distillation text/image-to-3D generation 3D-to-3D editing high-quality texture synthesis latent diffusion models text-to-3D initialization