Janus: Decoupling Visual Encoding for Unified Multimodal Understanding
  and Generation
In this paper, we introduce Janus, an autoregressive framework that unifies
multimodal understanding and generation. Prior research often relies on a
single visual encoder for both tasks, such as Chameleon. However, due to the
differing levels of information granularity required by multimodal
understanding and generation, this approach can lead to suboptimal performance,
particularly in multimodal understanding. To address this issue, we decouple
visual encoding into separate pathways, while still leveraging a single,
unified transformer architecture for processing. The decoupling not only
alleviates the conflict between the visual encoder's roles in understanding and
generation, but also enhances the framework's flexibility. For instance, both
the multimodal understanding and generation components can independently select
their most suitable encoding methods. Experiments show that Janus surpasses
previous unified model and matches or exceeds the performance of task-specific
models. The simplicity, high flexibility, and effectiveness of Janus make it a
strong candidate for next-generation unified multimodal models.
6711d3f2b6b1901abbf8c1f3
Chengyue Wu
claimed_verified
2024-10-18T08:55:10.857Z
617526c9de8feb54b0ce45ad
/avatars/7faf8c6f71fc318a0113d780d376c381.svg
Wu Chengyue
user
WuChengyue
6711d3f2b6b1901abbf8c1f4
Xiaokang Chen
claimed_verified
2025-02-05T15:55:59.335Z
6635e701420baf7bc3f93561
/avatars/cdbb3085fee73ac520888977e2c575ea.svg
Xiaokang Chen
user
CharlesCXK
6711d3f2b6b1901abbf8c1f5
Zhiyu Wu
6711d3f2b6b1901abbf8c1f6
Yiyang Ma
admin_assigned
2024-10-18T08:58:28.664Z
667282bf992bdf3ecb3bdc86
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/aXQRm8RUJXNJYZaBQzlDu.jpeg
Ma_Yiyang
user
CNMaxwell
6711d3f2b6b1901abbf8c1f7
Xingchao Liu
admin_assigned
2024-10-18T08:58:36.546Z
646b0bbdec9a61e871799339
https://cdn-avatars.huggingface.co/v1/production/uploads/646b0bbdec9a61e871799339/Xippv-ajkHkrGGAA7caLn.jpeg
Xingchao Liu
user
XCLiu
6711d3f2b6b1901abbf8c1f8
Zizheng Pan
admin_assigned
2024-10-18T08:58:43.853Z
629d55b53a3221bb2117422c
/avatars/e39f25cd7190d282478cdac20a7700ee.svg
Zizheng Pan
user
zizhpan
6711d3f2b6b1901abbf8c1f9
Wen Liu
claimed_verified
2024-12-30T19:38:22.611Z
63198e8802fb322037332f2d
/avatars/d3f9c206e387df35beb0ed0ef1cdf865.svg
Wen Liu
user
doubility123
6711d3f2b6b1901abbf8c1fa
Zhenda Xie
admin_assigned
2024-10-18T08:59:00.682Z
6539f6ea26df26ecd1393c37
https://cdn-avatars.huggingface.co/v1/production/uploads/6539f6ea26df26ecd1393c37/9VJusLLAiLhUxfBAFggpF.jpeg
Zhenda Xie
user
zdaxie
6711d3f2b6b1901abbf8c1fb
Xingkai Yu
6711d3f2b6b1901abbf8c1fc
Chong Ruan
admin_assigned
2024-10-18T08:59:22.321Z
6398203609f12714ed1935c2
https://cdn-avatars.huggingface.co/v1/production/uploads/6398203609f12714ed1935c2/uXgl0LgKnFYjq1Wz39-a6.jpeg
Chong Ruan
user
Chester111
6711d3f2b6b1901abbf8c1fd
Ping Luo
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding
  and Generation
In this paper, we introduce Janus, an autoregressive framework that unifies
multimodal understanding and generation. Prior research often relies on a
single visual encoder for both tasks, such as Chameleon. However, due to the
differing levels of information granularity required by multimodal
understanding and generation, this approach can lead to suboptimal performance,
particularly in multimodal understanding. To address this issue, we decouple
visual encoding into separate pathways, while still leveraging a single,
unified transformer architecture for processing. The decoupling not only
alleviates the conflict between the visual encoder's roles in understanding and
generation, but also enhances the framework's flexibility. For instance, both
the multimodal understanding and generation components can independently select
their most suitable encoding methods. Experiments show that Janus surpasses
previous unified model and matches or exceeds the performance of task-specific
models. The simplicity, high flexibility, and effectiveness of Janus make it a
strong candidate for next-generation unified multimodal models.
Janus, an autoregressive framework with separate visual encoding pathways within a unified transformer architecture, enhances performance in unified multimodal understanding and generation.
autoregressive framework multimodal understanding multimodal generation visual encoder transformer architecture encoding methods