EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via
  Multimodal LLM
Significant achievements in personalization of diffusion models have been
witnessed. Conventional tuning-free methods mostly encode multiple reference
images by averaging their image embeddings as the injection condition, but such
an image-independent operation cannot perform interaction among images to
capture consistent visual elements within multiple references. Although the
tuning-based Low-Rank Adaptation (LoRA) can effectively extract consistent
elements within multiple images through the training process, it necessitates
specific finetuning for each distinct image group. This paper introduces
EasyRef, a novel plug-and-play adaptation method that enables diffusion models
to be conditioned on multiple reference images and the text prompt. To
effectively exploit consistent visual elements within multiple images, we
leverage the multi-image comprehension and instruction-following capabilities
of the multimodal large language model (MLLM), prompting it to capture
consistent visual elements based on the instruction. Besides, injecting the
MLLM's representations into the diffusion process through adapters can easily
generalize to unseen domains, mining the consistent visual elements within
unseen data. To mitigate computational costs and enhance fine-grained detail
preservation, we introduce an efficient reference aggregation strategy and a
progressive training scheme. Finally, we introduce MRBench, a new
multi-reference image generation benchmark. Experimental results demonstrate
EasyRef surpasses both tuning-free methods like IP-Adapter and tuning-based
methods like LoRA, achieving superior aesthetic quality and robust zero-shot
generalization across diverse domains.
675bad005b45ed6ecd315666
Zhuofan Zong
claimed_verified
2024-12-16T09:42:15.087Z
64db05f23725f8d9a9f566c4
/avatars/345b6d2dfadb91ac1f02be023dd27c8f.svg
Zhuofan Zong
user
zongzhuofan
675bad005b45ed6ecd315667
Dongzhi Jiang
claimed_verified
2024-12-13T15:22:14.337Z
6349214f8146350b3a4c5cdf
/avatars/cfd24caac9a87efb528d0f4c375932bc.svg
Dongzhi Jiang
user
CaraJ
675bad005b45ed6ecd315668
Bingqi Ma
admin_assigned
2024-12-16T11:20:31.312Z
64424d1d5933b4560698ff86
/avatars/f44d73744796151eda0ef0035ad10a2b.svg
bingqi ma
user
A157801
675bad005b45ed6ecd315669
Guanglu Song
admin_assigned
2024-12-16T11:20:19.713Z
636a2efffbc31ee68fee5c1c
/avatars/21e119f23e1e6fe926567409e74d1a89.svg
Song
user
Guanglu
675bad005b45ed6ecd31566a
Hao Shao
admin_assigned
2024-12-16T11:20:07.328Z
651bda4337fecec1fe8bb09f
/avatars/f1b3d2b9874022e64d8e5dfeac0e7120.svg
Hao Shao
user
deepcs233
675bad005b45ed6ecd31566b
Dazhong Shen
admin_assigned
2024-12-16T11:20:00.987Z
645322c20446280667d6fce8
/avatars/776cf496f58f67bc2c80d9c6e5ff433e.svg
Dazhong Sehn
user
Dazhong
675bad005b45ed6ecd31566c
Yu Liu
675bad005b45ed6ecd31566d
Hongsheng Li
admin_assigned
2024-12-16T11:19:33.809Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via
  Multimodal LLM
Significant achievements in personalization of diffusion models have been
witnessed. Conventional tuning-free methods mostly encode multiple reference
images by averaging their image embeddings as the injection condition, but such
an image-independent operation cannot perform interaction among images to
capture consistent visual elements within multiple references. Although the
tuning-based Low-Rank Adaptation (LoRA) can effectively extract consistent
elements within multiple images through the training process, it necessitates
specific finetuning for each distinct image group. This paper introduces
EasyRef, a novel plug-and-play adaptation method that enables diffusion models
to be conditioned on multiple reference images and the text prompt. To
effectively exploit consistent visual elements within multiple images, we
leverage the multi-image comprehension and instruction-following capabilities
of the multimodal large language model (MLLM), prompting it to capture
consistent visual elements based on the instruction. Besides, injecting the
MLLM's representations into the diffusion process through adapters can easily
generalize to unseen domains, mining the consistent visual elements within
unseen data. To mitigate computational costs and enhance fine-grained detail
preservation, we introduce an efficient reference aggregation strategy and a
progressive training scheme. Finally, we introduce MRBench, a new
multi-reference image generation benchmark. Experimental results demonstrate
EasyRef surpasses both tuning-free methods like IP-Adapter and tuning-based
methods like LoRA, achieving superior aesthetic quality and robust zero-shot
generalization across diverse domains.
EasyRef enhances diffusion models for multi-reference image generation by leveraging multimodal large language models and efficient training strategies, outperforming existing methods in aesthetic quality and generalization.
diffusion models EasyRef Low-Rank Adaptation (LoRA) multimodal large language model (MLLM) reference aggregation progressive training MRBench IP-Adapter