PartGen: Part-level 3D Generation and Reconstruction with Multi-View
  Diffusion Models
Text- or image-to-3D generators and 3D scanners can now produce 3D assets
with high-quality shapes and textures. These assets typically consist of a
single, fused representation, like an implicit neural field, a Gaussian
mixture, or a mesh, without any useful structure. However, most applications
and creative workflows require assets to be made of several meaningful parts
that can be manipulated independently. To address this gap, we introduce
PartGen, a novel approach that generates 3D objects composed of meaningful
parts starting from text, an image, or an unstructured 3D object. First, given
multiple views of a 3D object, generated or rendered, a multi-view diffusion
model extracts a set of plausible and view-consistent part segmentations,
dividing the object into parts. Then, a second multi-view diffusion model takes
each part separately, fills in the occlusions, and uses those completed views
for 3D reconstruction by feeding them to a 3D reconstruction network. This
completion process considers the context of the entire object to ensure that
the parts integrate cohesively. The generative completion model can make up for
the information missing due to occlusions; in extreme cases, it can hallucinate
entirely invisible parts based on the input 3D asset. We evaluate our method on
generated and real 3D assets and show that it outperforms segmentation and
part-extraction baselines by a large margin. We also showcase downstream
applications such as 3D part editing.
676baa15295f85d93eb48f24
Minghao Chen
676baa15295f85d93eb48f25
Roman Shapovalov
676baa15295f85d93eb48f26
Iro Laina
676baa15295f85d93eb48f27
Tom Monnier
676baa15295f85d93eb48f28
Jianyuan Wang
676baa15295f85d93eb48f29
David Novotny
676baa15295f85d93eb48f2a
Andrea Vedaldi
PartGen: Part-level 3D Generation and Reconstruction with Multi-View
  Diffusion Models
Text- or image-to-3D generators and 3D scanners can now produce 3D assets
with high-quality shapes and textures. These assets typically consist of a
single, fused representation, like an implicit neural field, a Gaussian
mixture, or a mesh, without any useful structure. However, most applications
and creative workflows require assets to be made of several meaningful parts
that can be manipulated independently. To address this gap, we introduce
PartGen, a novel approach that generates 3D objects composed of meaningful
parts starting from text, an image, or an unstructured 3D object. First, given
multiple views of a 3D object, generated or rendered, a multi-view diffusion
model extracts a set of plausible and view-consistent part segmentations,
dividing the object into parts. Then, a second multi-view diffusion model takes
each part separately, fills in the occlusions, and uses those completed views
for 3D reconstruction by feeding them to a 3D reconstruction network. This
completion process considers the context of the entire object to ensure that
the parts integrate cohesively. The generative completion model can make up for
the information missing due to occlusions; in extreme cases, it can hallucinate
entirely invisible parts based on the input 3D asset. We evaluate our method on
generated and real 3D assets and show that it outperforms segmentation and
part-extraction baselines by a large margin. We also showcase downstream
applications such as 3D part editing.
PartGen generates 3D objects composed of meaningful parts from text, images, or unstructured 3D assets using multi-view diffusion models for segmentation and completion.
diffusion models part segmentations 3D reconstruction multi-view diffusion model 3D part editing