Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion
Existing single image-to-3D creation methods typically involve a two-stage
process, first generating multi-view images, and then using these images for 3D
reconstruction. However, training these two stages separately leads to
significant data bias in the inference phase, thus affecting the quality of
reconstructed results. We introduce a unified 3D generation framework, named
Ouroboros3D, which integrates diffusion-based multi-view image generation and
3D reconstruction into a recursive diffusion process. In our framework, these
two modules are jointly trained through a self-conditioning mechanism, allowing
them to adapt to each other's characteristics for robust inference. During the
multi-view denoising process, the multi-view diffusion model uses the 3D-aware
maps rendered by the reconstruction module at the previous timestep as
additional conditions. The recursive diffusion framework with 3D-aware feedback
unites the entire process and improves geometric consistency.Experiments show
that our framework outperforms separation of these two stages and existing
methods that combine them at the inference phase. Project page:
https://costwen.github.io/Ouroboros3D/
666119207427717d286d8ece
Hao Wen
claimed_verified
2024-08-05T09:46:29.864Z
6356194ce1e6a03a1ed3e0b3
/avatars/92adf0c54549ac99315fa36da78b0448.svg
wenhao
user
costwen
666119207427717d286d8ecf
Zehuan Huang
admin_assigned
2024-06-06T07:25:33.462Z
6375d136dee28348a9c63cbf
https://cdn-avatars.huggingface.co/v1/production/uploads/6375d136dee28348a9c63cbf/gK465HBrQWIOZ-qtHb-Vh.jpeg
zehuan-huang
user
huanngzh
666119207427717d286d8ed0
Yaohui Wang
admin_assigned
2024-06-06T07:25:22.841Z
63201256c6b20f03c829c4b8
/avatars/a42092119777d65e60b12eb5ba0e45f1.svg
Yaohui Wang
user
YaohuiW
666119207427717d286d8ed1
Xinyuan Chen
admin_assigned
2024-06-06T07:24:50.672Z
643e943a70c6a27621eb1c89
/avatars/73ec521ab5ba84cc7908c52c0acef6ef.svg
Xinyuan Chen
user
AriaChen
666119207427717d286d8ed2
Yu Qiao
666119207427717d286d8ed3
Lu Sheng
claimed_verified
2024-12-06T15:58:19.372Z
65b722dbe02a17f0f8d1cc6b
/avatars/65f20601ef9b8ebfdddadd737f9153d6.svg
Lu Sheng
user
lsheng2024
Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion
Existing single image-to-3D creation methods typically involve a two-stage
process, first generating multi-view images, and then using these images for 3D
reconstruction. However, training these two stages separately leads to
significant data bias in the inference phase, thus affecting the quality of
reconstructed results. We introduce a unified 3D generation framework, named
Ouroboros3D, which integrates diffusion-based multi-view image generation and
3D reconstruction into a recursive diffusion process. In our framework, these
two modules are jointly trained through a self-conditioning mechanism, allowing
them to adapt to each other's characteristics for robust inference. During the
multi-view denoising process, the multi-view diffusion model uses the 3D-aware
maps rendered by the reconstruction module at the previous timestep as
additional conditions. The recursive diffusion framework with 3D-aware feedback
unites the entire process and improves geometric consistency.Experiments show
that our framework outperforms separation of these two stages and existing
methods that combine them at the inference phase. Project page:
https://costwen.github.io/Ouroboros3D/
A unified framework, Ouroboros3D, combines diffusion-based multi-view image generation and 3D reconstruction through a recursive diffusion process, improving geometric consistency and reducing data bias.
diffusion-based multi-view image generation 3D reconstruction recursive diffusion process self-conditioning mechanism multi-view denoising 3D-aware maps geometric consistency