Ola: Pushing the Frontiers of Omni-Modal Language Model with Progressive
  Modality Alignment
Recent advances in large language models, particularly following GPT-4o, have
sparked increasing interest in developing omni-modal models capable of
understanding more modalities. While some open-source alternatives have
emerged, there is still a notable lag behind specialized single-modality models
in performance. In this paper, we present Ola, an Omni-modal language model
that achieves competitive performance across image, video, and audio
understanding compared to specialized counterparts. The core design of Ola lies
in its progressive modality alignment strategy that extends the supporting
modality of the language model progressively. Our training pipeline begins with
the most distinct modalities: image and text, then gradually expands the skill
sets of the model using speech data that connects language and audio knowledge,
and video data that connects all modalities. The progressive learning pipeline
also enables us to maintain a relatively small size of the cross-modal
alignment data, making developing omni-modal from existing vision-language
models easy and less costly. Moreover, to unlock an advanced interactive
experience like GPT-4o, we further design a sentence-wise decoding solution for
streaming speech generation. Extensive experiments demonstrate that Ola
surpasses existing open omni-modal LLMs across all modalities while achieving
highly competitive performance compared to state-of-the-art specialized models
of similar sizes. We aim to make Ola a fully open omni-modal understanding
solution to advance future research in this emerging field. Model weights,
code, and data are open-sourced at https://github.com/Ola-Omni/Ola.
67a586fad177de2eeba7de7b
Zuyan Liu
claimed_verified
2025-02-07T09:58:10.679Z
64f001bfabd9fb1914398bd5
https://cdn-avatars.huggingface.co/v1/production/uploads/64f001bfabd9fb1914398bd5/9teH82hkBI4csIz_WQh5q.jpeg
liuzuyan
user
Zuyan
67a586fad177de2eeba7de7c
Yuhao Dong
admin_assigned
2025-02-07T10:37:45.556Z
652965773a416e1f2173443b
https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg
Yuhao Dong
user
THUdyh
67a586fad177de2eeba7de7d
Jiahui Wang
67a586fad177de2eeba7de7e
Ziwei Liu
admin_assigned
2025-02-07T10:38:14.852Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
67a586fad177de2eeba7de7f
Winston Hu
admin_assigned
2025-02-07T10:38:23.468Z
63673bb9d0ee6e2662be0ec1
/avatars/1b8976785d64bc4e3f7159ccdb7f06c5.svg
Qingqiao Hu
user
WinstonHu
67a586fad177de2eeba7de80
Jiwen Lu
admin_assigned
2025-02-07T10:38:29.456Z
66c44203ea476bea05e9fcd7
/avatars/b061eebec609446e669f5ad6365959f9.svg
lu
user
jiwenlu
67a586fad177de2eeba7de81
Yongming Rao
admin_assigned
2025-02-07T10:38:35.766Z
63e4865354f51ea342d45d78
/avatars/2e7eccc878751331ca8b282f53e38899.svg
Yongming Rao
user
raoyongming
Ola: Pushing the Frontiers of Omni-Modal Language Model with Progressive
  Modality Alignment
Recent advances in large language models, particularly following GPT-4o, have
sparked increasing interest in developing omni-modal models capable of
understanding more modalities. While some open-source alternatives have
emerged, there is still a notable lag behind specialized single-modality models
in performance. In this paper, we present Ola, an Omni-modal language model
that achieves competitive performance across image, video, and audio
understanding compared to specialized counterparts. The core design of Ola lies
in its progressive modality alignment strategy that extends the supporting
modality of the language model progressively. Our training pipeline begins with
the most distinct modalities: image and text, then gradually expands the skill
sets of the model using speech data that connects language and audio knowledge,
and video data that connects all modalities. The progressive learning pipeline
also enables us to maintain a relatively small size of the cross-modal
alignment data, making developing omni-modal from existing vision-language
models easy and less costly. Moreover, to unlock an advanced interactive
experience like GPT-4o, we further design a sentence-wise decoding solution for
streaming speech generation. Extensive experiments demonstrate that Ola
surpasses existing open omni-modal LLMs across all modalities while achieving
highly competitive performance compared to state-of-the-art specialized models
of similar sizes. We aim to make Ola a fully open omni-modal understanding
solution to advance future research in this emerging field. Model weights,
code, and data are open-sourced at https://github.com/Ola-Omni/Ola.
Ola is an omni-modal language model that achieves competitive performance across image, video, and audio understanding through progressive modality alignment and efficient cross-modal training.
omni-modal language model progressive modality alignment cross-modal alignment vision-language models streaming speech generation