SEED-X: Multimodal Models with Unified Multi-granularity Comprehension
  and Generation
The rapid evolution of multimodal foundation model has demonstrated
significant progresses in vision-language understanding and generation, e.g.,
our previous work SEED-LLaMA. However, there remains a gap between its
capability and the real-world applicability, primarily due to the model's
limited capacity to effectively respond to various user instructions and
interact with diverse visual data. In this work, we focus on bridging this gap
through integrating two enhanced features: (1) comprehending images of
arbitrary sizes and ratios, and (2) enabling multi-granularity image
generation. We present a unified and versatile foundation model, namely,
SEED-X, which is able to model multi-granularity visual semantics for
comprehension and generation tasks. Besides the competitive results on public
benchmarks, SEED-X demonstrates its effectiveness in handling real-world
applications across various domains after instruction tuning. We hope that our
work will inspire future research into what can be achieved by versatile
multimodal foundation models in real-world applications. The models, codes, and
datasets will be released in https://github.com/AILab-CVC/SEED-X.
662719b80c759e8f21192d3c
Yuying Ge
admin_assigned
2024-04-23T08:17:07.027Z
6455cc8f654d8bccae50e4d4
/avatars/506a9992e5bf52e06d37cc22e4b307c0.svg
Yuying Ge
user
tttoaster
662719b80c759e8f21192d3d
Sijie Zhao
admin_assigned
2024-04-23T08:17:13.876Z
652cf6bad78452c474434fa6
/avatars/106bd7a6f8af3ae3b079a0533f6fb619.svg
Sijie Zhao
user
walkingshadow
662719b80c759e8f21192d3e
Jinguo Zhu
662719b80c759e8f21192d3f
Yixiao Ge
admin_assigned
2024-04-23T08:17:26.666Z
640e9762b03f4cd29f58d982
/avatars/81da37d628163fe3e094b247c7c3a3b5.svg
Yixiao Ge
user
yxgeee
662719b80c759e8f21192d40
Kun Yi
admin_assigned
2024-04-23T08:17:34.010Z
64071f662e309e65451fab0f
/avatars/3c1ad16a16b82fdc2618eeeaa75fffda.svg
kunyi
user
Kunyi
662719b80c759e8f21192d41
Lin Song
662719b80c759e8f21192d42
Chen Li
662719b80c759e8f21192d43
Xiaohan Ding
admin_assigned
2024-04-23T08:17:59.777Z
6564f049957ac688429a6754
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/nE6q5TlyLKlJpbxCKoDN8.png
xiaohanding
user
DingXiaoH
662719b80c759e8f21192d44
Ying Shan
admin_assigned
2024-04-23T08:18:17.587Z
63ca3ddc04c979828310bfcb
/avatars/615e0d8622950b4408b40d550f02a894.svg
Ying Shan
user
yshan2u
SEED-X: Multimodal Models with Unified Multi-granularity Comprehension
  and Generation
The rapid evolution of multimodal foundation model has demonstrated
significant progresses in vision-language understanding and generation, e.g.,
our previous work SEED-LLaMA. However, there remains a gap between its
capability and the real-world applicability, primarily due to the model's
limited capacity to effectively respond to various user instructions and
interact with diverse visual data. In this work, we focus on bridging this gap
through integrating two enhanced features: (1) comprehending images of
arbitrary sizes and ratios, and (2) enabling multi-granularity image
generation. We present a unified and versatile foundation model, namely,
SEED-X, which is able to model multi-granularity visual semantics for
comprehension and generation tasks. Besides the competitive results on public
benchmarks, SEED-X demonstrates its effectiveness in handling real-world
applications across various domains after instruction tuning. We hope that our
work will inspire future research into what can be achieved by versatile
multimodal foundation models in real-world applications. The models, codes, and
datasets will be released in https://github.com/AILab-CVC/SEED-X.
SEED-X is a versatile multimodal foundation model that enhances vision-language understanding and generation by comprehending diverse image sizes and enabling multi-granularity image generation.
multimodal foundation model vision-language understanding generation SEED-LLaMA SEED-X multi-granularity visual semantics