Taming Mode Collapse in Score Distillation for Text-to-3D Generation
Despite the remarkable performance of score distillation in text-to-3D
generation, such techniques notoriously suffer from view inconsistency issues,
also known as "Janus" artifact, where the generated objects fake each view with
multiple front faces. Although empirically effective methods have approached
this problem via score debiasing or prompt engineering, a more rigorous
perspective to explain and tackle this problem remains elusive. In this paper,
we reveal that the existing score distillation-based text-to-3D generation
frameworks degenerate to maximal likelihood seeking on each view independently
and thus suffer from the mode collapse problem, manifesting as the Janus
artifact in practice. To tame mode collapse, we improve score distillation by
re-establishing in entropy term in the corresponding variational objective,
which is applied to the distribution of rendered images. Maximizing the entropy
encourages diversity among different views in generated 3D assets, thereby
mitigating the Janus problem. Based on this new objective, we derive a new
update rule for 3D score distillation, dubbed Entropic Score Distillation
(ESD). We theoretically reveal that ESD can be simplified and implemented by
just adopting the classifier-free guidance trick upon variational score
distillation. Although embarrassingly straightforward, our extensive
experiments successfully demonstrate that ESD can be an effective treatment for
Janus artifacts in score distillation.
6594cabd315340de5ff0d1d2
Peihao Wang
claimed_verified
2024-02-09T09:28:32.291Z
62c8f1773e28b8ee4cddca09
/avatars/9edbcc84e4ab2b335011daab8acfdbb8.svg
Peihao Wang
user
peihaowang
6594cabd315340de5ff0d1d3
Dejia Xu
claimed_verified
2024-06-03T07:06:09.797Z
62f687a0c58915315c4ff75d
/avatars/b657180c7666735062782edd4f6a69c9.svg
Dejia Xu
user
ir1d
6594cabd315340de5ff0d1d4
Zhiwen Fan
6594cabd315340de5ff0d1d5
Dilin Wang
6594cabd315340de5ff0d1d6
Sreyas Mohan
6594cabd315340de5ff0d1d7
Forrest Iandola
6594cabd315340de5ff0d1d8
Rakesh Ranjan
6594cabd315340de5ff0d1d9
Yilei Li
6594cabd315340de5ff0d1da
Qiang Liu
6594cabd315340de5ff0d1db
Zhangyang Wang
6594cabd315340de5ff0d1dc
Vikas Chandra
Taming Mode Collapse in Score Distillation for Text-to-3D Generation
Despite the remarkable performance of score distillation in text-to-3D
generation, such techniques notoriously suffer from view inconsistency issues,
also known as "Janus" artifact, where the generated objects fake each view with
multiple front faces. Although empirically effective methods have approached
this problem via score debiasing or prompt engineering, a more rigorous
perspective to explain and tackle this problem remains elusive. In this paper,
we reveal that the existing score distillation-based text-to-3D generation
frameworks degenerate to maximal likelihood seeking on each view independently
and thus suffer from the mode collapse problem, manifesting as the Janus
artifact in practice. To tame mode collapse, we improve score distillation by
re-establishing in entropy term in the corresponding variational objective,
which is applied to the distribution of rendered images. Maximizing the entropy
encourages diversity among different views in generated 3D assets, thereby
mitigating the Janus problem. Based on this new objective, we derive a new
update rule for 3D score distillation, dubbed Entropic Score Distillation
(ESD). We theoretically reveal that ESD can be simplified and implemented by
just adopting the classifier-free guidance trick upon variational score
distillation. Although embarrassingly straightforward, our extensive
experiments successfully demonstrate that ESD can be an effective treatment for
Janus artifacts in score distillation.
Maximizing entropy in score distillation improves text-to-3D generation by addressing view inconsistency issues and mode collapse via a method called Entropic Score Distillation.
score distillation text-to-3D generation Janus artifact mode collapse entropy term variational objective Entropic Score Distillation classifier-free guidance trick