Phidias: A Generative Model for Creating 3D Content from Text, Image,
  and 3D Conditions with Reference-Augmented Diffusion
In 3D modeling, designers often use an existing 3D model as a reference to
create new ones. This practice has inspired the development of Phidias, a novel
generative model that uses diffusion for reference-augmented 3D generation.
Given an image, our method leverages a retrieved or user-provided 3D reference
model to guide the generation process, thereby enhancing the generation
quality, generalization ability, and controllability. Our model integrates
three key components: 1) meta-ControlNet that dynamically modulates the
conditioning strength, 2) dynamic reference routing that mitigates misalignment
between the input image and 3D reference, and 3) self-reference augmentations
that enable self-supervised training with a progressive curriculum.
Collectively, these designs result in a clear improvement over existing
methods. Phidias establishes a unified framework for 3D generation using text,
image, and 3D conditions with versatile applications.
66ea3def1e62ad9403ce2075
Zhenwei Wang
claimed_verified
2024-09-18T07:01:27.359Z
6305e9471e731505a76b9099
https://cdn-avatars.huggingface.co/v1/production/uploads/6305e9471e731505a76b9099/w9csjXIEUzrd0sa1FxIXm.jpeg
Zhenwei WANG (Zavier)
user
ZhenweiWang
66ea3def1e62ad9403ce2076
Tengfei Wang
admin_assigned
2024-09-18T07:55:49.095Z
62e7c26236a8e8a827ff0891
/avatars/15648407233bf5757262dbc04464d02e.svg
Tengfei Wang
user
tfwang
66ea3def1e62ad9403ce2077
Zexin He
admin_assigned
2024-09-18T07:55:54.725Z
63a5d18a8fb23d08bb30094c
https://cdn-avatars.huggingface.co/v1/production/uploads/1671811420431-noauth.png
Zexin He
user
zxhezexin
66ea3def1e62ad9403ce2078
Gerhard Hancke
66ea3def1e62ad9403ce2079
Ziwei Liu
admin_assigned
2024-09-18T07:55:18.257Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
66ea3def1e62ad9403ce207a
Rynson W. H. Lau
Phidias: A Generative Model for Creating 3D Content from Text, Image,
  and 3D Conditions with Reference-Augmented Diffusion
In 3D modeling, designers often use an existing 3D model as a reference to
create new ones. This practice has inspired the development of Phidias, a novel
generative model that uses diffusion for reference-augmented 3D generation.
Given an image, our method leverages a retrieved or user-provided 3D reference
model to guide the generation process, thereby enhancing the generation
quality, generalization ability, and controllability. Our model integrates
three key components: 1) meta-ControlNet that dynamically modulates the
conditioning strength, 2) dynamic reference routing that mitigates misalignment
between the input image and 3D reference, and 3) self-reference augmentations
that enable self-supervised training with a progressive curriculum.
Collectively, these designs result in a clear improvement over existing
methods. Phidias establishes a unified framework for 3D generation using text,
image, and 3D conditions with versatile applications.
Phidias, a diffusion-based generative model, uses a 3D reference to improve 3D generation quality, generalization, and controllability through meta-ControlNet, dynamic reference routing, and self-reference augmentations.
diffusion meta-ControlNet dynamic reference routing self-reference augmentations 3D generation 3D reference text image conditions