MV-Adapter: Multi-view Consistent Image Generation Made Easy
Existing multi-view image generation methods often make invasive
modifications to pre-trained text-to-image (T2I) models and require full
fine-tuning, leading to (1) high computational costs, especially with large
base models and high-resolution images, and (2) degradation in image quality
due to optimization difficulties and scarce high-quality 3D data. In this
paper, we propose the first adapter-based solution for multi-view image
generation, and introduce MV-Adapter, a versatile plug-and-play adapter that
enhances T2I models and their derivatives without altering the original network
structure or feature space. By updating fewer parameters, MV-Adapter enables
efficient training and preserves the prior knowledge embedded in pre-trained
models, mitigating overfitting risks. To efficiently model the 3D geometric
knowledge within the adapter, we introduce innovative designs that include
duplicated self-attention layers and parallel attention architecture, enabling
the adapter to inherit the powerful priors of the pre-trained models to model
the novel 3D knowledge. Moreover, we present a unified condition encoder that
seamlessly integrates camera parameters and geometric information, facilitating
applications such as text- and image-based 3D generation and texturing.
MV-Adapter achieves multi-view generation at 768 resolution on Stable Diffusion
XL (SDXL), and demonstrates adaptability and versatility. It can also be
extended to arbitrary view generation, enabling broader applications. We
demonstrate that MV-Adapter sets a new quality standard for multi-view image
generation, and opens up new possibilities due to its efficiency, adaptability
and versatility.
6752727b61e5a1edbc6f45fc
Zehuan Huang
admin_assigned
2024-12-06T10:41:33.167Z
6375d136dee28348a9c63cbf
https://cdn-avatars.huggingface.co/v1/production/uploads/6375d136dee28348a9c63cbf/gK465HBrQWIOZ-qtHb-Vh.jpeg
zehuan-huang
user
huanngzh
6752727b61e5a1edbc6f45fd
Yuan-Chen Guo
admin_assigned
2024-12-06T10:41:38.686Z
6346aaa3f06b237ba4e297b0
/avatars/5acb986e993eab1461200f3e9d99d022.svg
Yuan-Chen Guo
user
bennyguo
6752727b61e5a1edbc6f45fe
Haoran Wang
6752727b61e5a1edbc6f45ff
Ran Yi
6752727b61e5a1edbc6f4600
Lizhuang Ma
6752727b61e5a1edbc6f4601
Yan-Pei Cao
admin_assigned
2024-12-06T10:42:23.272Z
638066faf022c8a5803f7eb8
/avatars/4cfd699c3f6c5461b12b7dc5e3fe183d.svg
Yanpei Cao
user
pookiefoof
6752727b61e5a1edbc6f4602
Lu Sheng
claimed_verified
2024-12-06T15:57:15.578Z
65b722dbe02a17f0f8d1cc6b
/avatars/65f20601ef9b8ebfdddadd737f9153d6.svg
Lu Sheng
user
lsheng2024
MV-Adapter: Multi-view Consistent Image Generation Made Easy
Existing multi-view image generation methods often make invasive
modifications to pre-trained text-to-image (T2I) models and require full
fine-tuning, leading to (1) high computational costs, especially with large
base models and high-resolution images, and (2) degradation in image quality
due to optimization difficulties and scarce high-quality 3D data. In this
paper, we propose the first adapter-based solution for multi-view image
generation, and introduce MV-Adapter, a versatile plug-and-play adapter that
enhances T2I models and their derivatives without altering the original network
structure or feature space. By updating fewer parameters, MV-Adapter enables
efficient training and preserves the prior knowledge embedded in pre-trained
models, mitigating overfitting risks. To efficiently model the 3D geometric
knowledge within the adapter, we introduce innovative designs that include
duplicated self-attention layers and parallel attention architecture, enabling
the adapter to inherit the powerful priors of the pre-trained models to model
the novel 3D knowledge. Moreover, we present a unified condition encoder that
seamlessly integrates camera parameters and geometric information, facilitating
applications such as text- and image-based 3D generation and texturing.
MV-Adapter achieves multi-view generation at 768 resolution on Stable Diffusion
XL (SDXL), and demonstrates adaptability and versatility. It can also be
extended to arbitrary view generation, enabling broader applications. We
demonstrate that MV-Adapter sets a new quality standard for multi-view image
generation, and opens up new possibilities due to its efficiency, adaptability
and versatility.
MV-Adapter, an adapter-based solution, enhances text-to-image models for multi-view image generation with efficient training, preserving pre-trained knowledge and facilitating the integration of 3D data.
text-to-image (T2I) models adapter-based solution MV-Adapter self-attention layers parallel attention architecture condition encoder Stable Diffusion XL (SDXL) multi-view image generation arbitrary view generation