BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture,
  Training and Dataset
Unifying image understanding and generation has gained growing attention in
recent research on multimodal models. Although design choices for image
understanding have been extensively studied, the optimal model architecture and
training recipe for a unified framework with image generation remain
underexplored. Motivated by the strong potential of autoregressive and
diffusion models for high-quality generation and scalability, we conduct a
comprehensive study of their use in unified multimodal settings, with emphasis
on image representations, modeling objectives, and training strategies.
Grounded in these investigations, we introduce a novel approach that employs a
diffusion transformer to generate semantically rich CLIP image features, in
contrast to conventional VAE-based representations. This design yields both
higher training efficiency and improved generative quality. Furthermore, we
demonstrate that a sequential pretraining strategy for unified models-first
training on image understanding and subsequently on image generation-offers
practical advantages by preserving image understanding capability while
developing strong image generation ability. Finally, we carefully curate a
high-quality instruction-tuning dataset BLIP3o-60k for image generation by
prompting GPT-4o with a diverse set of captions covering various scenes,
objects, human gestures, and more. Building on our innovative model design,
training recipe, and datasets, we develop BLIP3-o, a suite of state-of-the-art
unified multimodal models. BLIP3-o achieves superior performance across most of
the popular benchmarks spanning both image understanding and generation tasks.
To facilitate future research, we fully open-source our models, including code,
model weights, training scripts, and pretraining and instruction tuning
datasets.
68254419181d43c25d829239
Jiuhai Chen
admin_assigned
2025-05-15T13:48:28.916Z
6393847e3e30234ae798b7be
/avatars/daeb8c37dff4432d837a69b87c196521.svg
JiuhaiChen
user
jiuhai
68254419181d43c25d82923a
Zhiyang Xu
admin_assigned
2025-05-15T13:48:51.984Z
64b6c686cf5117d7962d8f62
/avatars/96ed7a9602aa4c21b3a3d89608e76dc8.svg
Zhiyang Xu
user
Zhiyang03
68254419181d43c25d82923b
Xichen Pan
claimed_verified
2025-05-15T10:31:43.038Z
63172831c92fd6fee3181f50
/avatars/0f57068a138cb181e9451bfc1ed3d1c0.svg
Xichen Pan
user
xcpan
68254419181d43c25d82923c
Yushi Hu
admin_assigned
2025-05-15T13:49:05.178Z
62b1474bdcbad6848a91a54e
/avatars/d7308899b46232cad4a48a0e876449a8.svg
Yushi Hu
user
yushihu
68254419181d43c25d82923d
Can Qin
68254419181d43c25d82923e
Tom Goldstein
admin_assigned
2025-05-15T13:49:13.762Z
6381ca7d65dc156aba0b933d
/avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
Tom Goldstein
user
tomgoldstein
68254419181d43c25d82923f
Lifu Huang
68254419181d43c25d829240
Tianyi Zhou
claimed_verified
2025-05-15T10:32:05.507Z
647f5af5b0e96764589f3b2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
Tianyi Zhou
user
zhoutianyi
68254419181d43c25d829241
Saining Xie
admin_assigned
2025-05-15T13:49:28.644Z
6596422646624a86ff3b3bda
/avatars/216e12b77e45ac5f1fa20932f5745411.svg
Saining Xie
user
sainx
68254419181d43c25d829242
Silvio Savarese
admin_assigned
2025-05-15T13:49:36.341Z
67d5674bbc03ef961e733ddd
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/3EUXNd-mKvsXFDlr1FETh.png
Silvio Savarese
user
SilvioSav8
68254419181d43c25d829243
Le Xue
admin_assigned
2025-05-15T13:49:58.945Z
63dd73e7422ca8d7f7e3698c
/avatars/7b0f8419f6941230b81dbbbb4f273edf.svg
Le Xue
user
SFXX
68254419181d43c25d829244
Caiming Xiong
admin_assigned
2025-05-15T13:49:42.774Z
649dbcc4e0fff1ed099dc80a
/avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
Caiming Xiong
user
cxiong
68254419181d43c25d829245
Ran Xu
claimed_verified
2025-05-15T10:31:39.465Z
6465c4c863e7e09dd02e3e1b
/avatars/200b029184d2616f98296a2c212f0785.svg
Ran Xu
user
xurantju
BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture,
  Training and Dataset
Unifying image understanding and generation has gained growing attention in
recent research on multimodal models. Although design choices for image
understanding have been extensively studied, the optimal model architecture and
training recipe for a unified framework with image generation remain
underexplored. Motivated by the strong potential of autoregressive and
diffusion models for high-quality generation and scalability, we conduct a
comprehensive study of their use in unified multimodal settings, with emphasis
on image representations, modeling objectives, and training strategies.
Grounded in these investigations, we introduce a novel approach that employs a
diffusion transformer to generate semantically rich CLIP image features, in
contrast to conventional VAE-based representations. This design yields both
higher training efficiency and improved generative quality. Furthermore, we
demonstrate that a sequential pretraining strategy for unified models-first
training on image understanding and subsequently on image generation-offers
practical advantages by preserving image understanding capability while
developing strong image generation ability. Finally, we carefully curate a
high-quality instruction-tuning dataset BLIP3o-60k for image generation by
prompting GPT-4o with a diverse set of captions covering various scenes,
objects, human gestures, and more. Building on our innovative model design,
training recipe, and datasets, we develop BLIP3-o, a suite of state-of-the-art
unified multimodal models. BLIP3-o achieves superior performance across most of
the popular benchmarks spanning both image understanding and generation tasks.
To facilitate future research, we fully open-source our models, including code,
model weights, training scripts, and pretraining and instruction tuning
datasets.
A diffusion transformer is used in a unified multimodal model framework to improve image generation while maintaining image understanding capabilities.
diffusion transformer CLIP image features VAE-based representations sequential pretraining image understanding image generation BLIP3-o instruction-tuning dataset