DreamComposer: Controllable 3D Object Generation via Multi-View
  Conditions
Utilizing pre-trained 2D large-scale generative models, recent works are
capable of generating high-quality novel views from a single in-the-wild image.
However, due to the lack of information from multiple views, these works
encounter difficulties in generating controllable novel views. In this paper,
we present DreamComposer, a flexible and scalable framework that can enhance
existing view-aware diffusion models by injecting multi-view conditions.
Specifically, DreamComposer first uses a view-aware 3D lifting module to obtain
3D representations of an object from multiple views. Then, it renders the
latent features of the target view from 3D representations with the multi-view
feature fusion module. Finally the target view features extracted from
multi-view inputs are injected into a pre-trained diffusion model. Experiments
show that DreamComposer is compatible with state-of-the-art diffusion models
for zero-shot novel view synthesis, further enhancing them to generate
high-fidelity novel view images with multi-view conditions, ready for
controllable 3D object reconstruction and various other applications.
657140ccb6f7eb7f31177890
Yunhan Yang
admin_assigned
2023-12-07T12:12:25.009Z
64b4eecf2fc8324fcb63b404
https://cdn-avatars.huggingface.co/v1/production/uploads/64b4eecf2fc8324fcb63b404/zGYqYVB4-o-GBMybJ8CDA.png
Yunhan Yang
user
yhyang-myron
657140ccb6f7eb7f31177891
Yukun Huang
claimed_verified
2024-09-26T08:26:39.818Z
638ee900ee7e45e0474a5712
https://cdn-avatars.huggingface.co/v1/production/uploads/638ee900ee7e45e0474a5712/KLli_eCbWwffKR7oLDmV3.jpeg
Yukun Huang
user
KevinHuang
657140ccb6f7eb7f31177892
Xiaoyang Wu
admin_assigned
2023-12-07T12:14:12.843Z
643e5d6a1d0e956d94bb3608
https://cdn-avatars.huggingface.co/v1/production/uploads/643e5d6a1d0e956d94bb3608/mTnu7Dts2RmDklHlp9Gqu.jpeg
Xiaoyang Wu
user
Gofinge
657140ccb6f7eb7f31177893
Yuan-Chen Guo
admin_assigned
2023-12-07T12:14:19.747Z
6346aaa3f06b237ba4e297b0
/avatars/5acb986e993eab1461200f3e9d99d022.svg
Yuan-Chen Guo
user
bennyguo
657140ccb6f7eb7f31177894
Song-Hai Zhang
657140ccb6f7eb7f31177895
Hengshuang Zhao
657140ccb6f7eb7f31177896
Tong He
admin_assigned
2023-12-07T12:15:14.826Z
63184606ab1bf092d6f059be
/avatars/b49d125d112a01106b48a2fcda481c3e.svg
Tong He
user
hetong007
657140ccb6f7eb7f31177897
Xihui Liu
claimed_verified
2024-06-03T18:30:14.921Z
65d5ec74cd05bc1eaa125040
/avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
Xihui Liu
user
XihuiLiu
DreamComposer: Controllable 3D Object Generation via Multi-View
  Conditions
Utilizing pre-trained 2D large-scale generative models, recent works are
capable of generating high-quality novel views from a single in-the-wild image.
However, due to the lack of information from multiple views, these works
encounter difficulties in generating controllable novel views. In this paper,
we present DreamComposer, a flexible and scalable framework that can enhance
existing view-aware diffusion models by injecting multi-view conditions.
Specifically, DreamComposer first uses a view-aware 3D lifting module to obtain
3D representations of an object from multiple views. Then, it renders the
latent features of the target view from 3D representations with the multi-view
feature fusion module. Finally the target view features extracted from
multi-view inputs are injected into a pre-trained diffusion model. Experiments
show that DreamComposer is compatible with state-of-the-art diffusion models
for zero-shot novel view synthesis, further enhancing them to generate
high-fidelity novel view images with multi-view conditions, ready for
controllable 3D object reconstruction and various other applications.
DreamComposer enhances diffusion models for novel view synthesis by integrating multi-view conditions to improve image quality and controllability.
view-aware diffusion models 3D lifting module multi-view feature fusion module novel view synthesis high-fidelity novel view images 3D object reconstruction