X-Fusion: Introducing New Modality to Frozen Large Language Models
We propose X-Fusion, a framework that extends pretrained Large Language
Models (LLMs) for multimodal tasks while preserving their language
capabilities. X-Fusion employs a dual-tower design with modality-specific
weights, keeping the LLM's parameters frozen while integrating vision-specific
information for both understanding and generation. Our experiments demonstrate
that X-Fusion consistently outperforms alternative architectures on both
image-to-text and text-to-image tasks. We find that incorporating
understanding-focused data improves generation quality, reducing image data
noise enhances overall performance, and feature alignment accelerates
convergence for smaller models but has minimal impact on larger ones. Our
findings provide valuable insights into building efficient unified multimodal
models.
6811c55384adfa26b82abd76
Sicheng Mo
admin_assigned
2025-04-30T12:22:58.085Z
637c94d3f219c71f93eda9ad
/avatars/6dae0c30755196ccc0a5a06b3981c47f.svg
Sicheng Mo
user
Sichengmo
6811c55384adfa26b82abd77
Thao Nguyen
6811c55384adfa26b82abd78
Xun Huang
claimed_verified
2025-06-12T12:44:09.735Z
6732bfed9e05f3e257315c8e
/avatars/abe9aa6914feef77b7205fe030e5bcc5.svg
Xun Huang
user
xunhuang1995
6811c55384adfa26b82abd79
Siddharth Srinivasan Iyer
6811c55384adfa26b82abd7a
Yijun Li
6811c55384adfa26b82abd7b
Yuchen Liu
6811c55384adfa26b82abd7c
Abhishek Tandon
6811c55384adfa26b82abd7d
Eli Shechtman
6811c55384adfa26b82abd7e
Krishna Kumar Singh
6811c55384adfa26b82abd7f
Yong Jae Lee
admin_assigned
2025-04-30T12:23:52.478Z
649f41ee70a478f8b36b2984
/avatars/1c9a76717a450ac4aeb62a1e823d2e4a.svg
Yong Jae Lee
user
yjlee0222
6811c55384adfa26b82abd80
Bolei Zhou
admin_assigned
2025-04-30T12:23:58.680Z
64aedf11c693a19547999a59
/avatars/83a54b7c217168c91cea15dd000f4af2.svg
Bolei Zhou
user
zhoubolei
6811c55384adfa26b82abd81
Yuheng Li
X-Fusion: Introducing New Modality to Frozen Large Language Models
We propose X-Fusion, a framework that extends pretrained Large Language
Models (LLMs) for multimodal tasks while preserving their language
capabilities. X-Fusion employs a dual-tower design with modality-specific
weights, keeping the LLM's parameters frozen while integrating vision-specific
information for both understanding and generation. Our experiments demonstrate
that X-Fusion consistently outperforms alternative architectures on both
image-to-text and text-to-image tasks. We find that incorporating
understanding-focused data improves generation quality, reducing image data
noise enhances overall performance, and feature alignment accelerates
convergence for smaller models but has minimal impact on larger ones. Our
findings provide valuable insights into building efficient unified multimodal
models.
X-Fusion enhances pretrained LLMs for multimodal tasks by integrating vision-specific information through a dual-tower design and freezes the LLM's parameters for consistency in understanding and generation.
Large Language Models (LLMs) multimodal tasks dual-tower design modality-specific weights vision-specific information image-to-text text-to-image understanding-focused data image data noise feature alignment