SynerGen-VL: Towards Synergistic Image Understanding and Generation with
  Vision Experts and Token Folding
The remarkable success of Large Language Models (LLMs) has extended to the
multimodal domain, achieving outstanding performance in image understanding and
generation. Recent efforts to develop unified Multimodal Large Language Models
(MLLMs) that integrate these capabilities have shown promising results.
However, existing approaches often involve complex designs in model
architecture or training pipeline, increasing the difficulty of model training
and scaling. In this paper, we propose SynerGen-VL, a simple yet powerful
encoder-free MLLM capable of both image understanding and generation. To
address challenges identified in existing encoder-free unified MLLMs, we
introduce the token folding mechanism and the vision-expert-based progressive
alignment pretraining strategy, which effectively support high-resolution image
understanding while reducing training complexity. After being trained on
large-scale mixed image-text data with a unified next-token prediction
objective, SynerGen-VL achieves or surpasses the performance of existing
encoder-free unified MLLMs with comparable or smaller parameter sizes, and
narrows the gap with task-specific state-of-the-art models, highlighting a
promising path toward future unified MLLMs. Our code and models shall be
released.
675d657d1c375f21ff6c008f
Hao Li
claimed_verified
2024-12-16T09:41:44.798Z
64b9033777ae61bcc80aa4f3
/avatars/408c335395c79f3df69fd9bf70abc312.svg
Hao Li
user
cpsxhao
675d657d1c375f21ff6c0090
Changyao Tian
admin_assigned
2024-12-16T10:23:50.435Z
64b7475efa7eabaae5f7ba94
https://cdn-avatars.huggingface.co/v1/production/uploads/64b7475efa7eabaae5f7ba94/YLv44PZM6tw1sxACY1-U_.png
Changyao Tian
user
Changyao
675d657d1c375f21ff6c0091
Jie Shao
admin_assigned
2024-12-16T10:24:24.661Z
644a2bccd9a3ae834104b065
/avatars/ee2caf787796cca438349d10089bdfd1.svg
Jie Shao
user
hehesang
675d657d1c375f21ff6c0092
Xizhou Zhu
admin_assigned
2024-12-16T10:24:30.898Z
64ae2359179421d320b1694b
/avatars/c387a75191005bcaa473091de5383a10.svg
Xizhou Zhu
user
Einsiedler
675d657d1c375f21ff6c0093
Zhaokai Wang
claimed_verified
2024-12-16T09:41:42.930Z
665d4b515fdfe8f923e347a7
/avatars/d114b24c02dadfca0a8aee104755a8ec.svg
Zhaokai Wang
user
wzk1015
675d657d1c375f21ff6c0094
Jinguo Zhu
675d657d1c375f21ff6c0095
Wenhan Dou
admin_assigned
2024-12-16T10:24:50.436Z
66efe658de163a536aa84178
/avatars/de547ed48ca37694e08ee926aea03d9f.svg
dou wenhan
user
douwh
675d657d1c375f21ff6c0096
Xiaogang Wang
admin_assigned
2024-12-16T10:25:01.045Z
66149831f9e58fe02b08f10e
/avatars/493636f2368e689c9241e53bc2e7b97e.svg
wangxiaogang
user
wangxiaogang
675d657d1c375f21ff6c0097
Hongsheng Li
admin_assigned
2024-12-16T10:25:17.271Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
675d657d1c375f21ff6c0098
Lewei Lu
admin_assigned
2024-12-16T10:25:24.403Z
65ead3ea908526a39082e641
/avatars/dcf870695fd56b06ca03d82f831e9019.svg
Lewei Lu
user
luotto
675d657d1c375f21ff6c0099
Jifeng Dai
admin_assigned
2024-12-16T10:25:31.520Z
64686f7172d9180d4ac8b4e4
/avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
Jifeng Dai
user
daijifeng
SynerGen-VL: Towards Synergistic Image Understanding and Generation with
  Vision Experts and Token Folding
The remarkable success of Large Language Models (LLMs) has extended to the
multimodal domain, achieving outstanding performance in image understanding and
generation. Recent efforts to develop unified Multimodal Large Language Models
(MLLMs) that integrate these capabilities have shown promising results.
However, existing approaches often involve complex designs in model
architecture or training pipeline, increasing the difficulty of model training
and scaling. In this paper, we propose SynerGen-VL, a simple yet powerful
encoder-free MLLM capable of both image understanding and generation. To
address challenges identified in existing encoder-free unified MLLMs, we
introduce the token folding mechanism and the vision-expert-based progressive
alignment pretraining strategy, which effectively support high-resolution image
understanding while reducing training complexity. After being trained on
large-scale mixed image-text data with a unified next-token prediction
objective, SynerGen-VL achieves or surpasses the performance of existing
encoder-free unified MLLMs with comparable or smaller parameter sizes, and
narrows the gap with task-specific state-of-the-art models, highlighting a
promising path toward future unified MLLMs. Our code and models shall be
released.
SynerGen-VL, an encoder-free MLLM using token folding and vision-expert-based progressive alignment, performs comparably to task-specific models on image understanding and generation with reduced training complexity.
Large Language Models Multimodal Large Language Models SynerGen-VL token folding mechanism vision-expert-based progressive alignment pretraining strategy high-resolution image understanding unified next-token prediction objective