MIO: A Foundation Model on Multimodal Tokens
In this paper, we introduce MIO, a novel foundation model built on multimodal
tokens, capable of understanding and generating speech, text, images, and
videos in an end-to-end, autoregressive manner. While the emergence of large
language models (LLMs) and multimodal large language models (MM-LLMs) propels
advancements in artificial general intelligence through their versatile
capabilities, they still lack true any-to-any understanding and generation.
Recently, the release of GPT-4o has showcased the remarkable potential of
any-to-any LLMs for complex real-world tasks, enabling omnidirectional input
and output across images, speech, and text. However, it is closed-source and
does not support the generation of multimodal interleaved sequences. To address
this gap, we present MIO, which is trained on a mixture of discrete tokens
across four modalities using causal multimodal modeling. MIO undergoes a
four-stage training process: (1) alignment pre-training, (2) interleaved
pre-training, (3) speech-enhanced pre-training, and (4) comprehensive
supervised fine-tuning on diverse textual, visual, and speech tasks. Our
experimental results indicate that MIO exhibits competitive, and in some cases
superior, performance compared to previous dual-modal baselines, any-to-any
model baselines, and even modality-specific baselines. Moreover, MIO
demonstrates advanced capabilities inherent to its any-to-any feature, such as
interleaved video-text generation, chain-of-visual-thought reasoning, visual
guideline generation, instructional image editing, etc.
66fa19f693b99e4c7c8a02ce
Zekun Wang
claimed_verified
2024-10-07T07:30:21.748Z
6149a9e95347647e6bb68882
https://cdn-avatars.huggingface.co/v1/production/uploads/6149a9e95347647e6bb68882/Jddln1FxScCeVgTSCNBpr.png
Zekun Moore Wang
user
ZenMoore
66fa19f693b99e4c7c8a02cf
King Zhu
claimed_verified
2024-10-18T09:25:03.209Z
6578265ddea7e2122d02f6ba
https://cdn-avatars.huggingface.co/v1/production/uploads/6578265ddea7e2122d02f6ba/Bh6JjoVF5ceLSjV7Z7nTk.jpeg
kang zhu
user
kangz
66fa19f693b99e4c7c8a02d0
Chunpu Xu
admin_assigned
2024-09-30T12:26:31.334Z
6580333b64bab467b9bcb615
/avatars/753afae2fefc748791550c733aa2c80b.svg
Chunpu Xu
user
cpxu
66fa19f693b99e4c7c8a02d1
Wangchunshu Zhou
admin_assigned
2024-09-30T12:26:42.767Z
628c8598ef14f971b698107f
/avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
Zhou
user
Wangchunshu
66fa19f693b99e4c7c8a02d2
Jiaheng Liu
admin_assigned
2024-09-30T12:26:48.968Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
66fa19f693b99e4c7c8a02d3
Yibo Zhang
66fa19f693b99e4c7c8a02d4
Jiashuo Wang
66fa19f693b99e4c7c8a02d5
Ning Shi
admin_assigned
2024-09-30T12:28:12.312Z
64e2f812bbee0c24b93fdb88
https://cdn-avatars.huggingface.co/v1/production/uploads/64e2f812bbee0c24b93fdb88/UdeMJFDWjRjydJ38WjnnO.jpeg
Ning Shi
user
MrShininnnnn
66fa19f693b99e4c7c8a02d6
Siyu Li
66fa19f693b99e4c7c8a02d7
Yizhi Li
claimed_verified
2025-07-25T08:40:23.287Z
6382252f54421460665ec501
https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
Yizhi Li
user
yizhilll
66fa19f693b99e4c7c8a02d8
Haoran Que
66fa19f693b99e4c7c8a02d9
Zhaoxiang Zhang
admin_assigned
2024-09-30T12:29:05.696Z
6350d59989def14ad21e11b3
/avatars/9a7ae6131e0a89f3461f267ee844b218.svg
z
user
smallbert
66fa19f693b99e4c7c8a02da
Yuanxing Zhang
admin_assigned
2024-09-30T12:29:51.984Z
60ff5c28aa025227d344b734
https://cdn-avatars.huggingface.co/v1/production/uploads/1627347980032-noauth.jpeg
Yuanxing Zhang
user
Onmyoji
66fa19f693b99e4c7c8a02db
Ge Zhang
admin_assigned
2024-09-30T12:30:04.866Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
66fa19f693b99e4c7c8a02dc
Ke Xu
66fa19f693b99e4c7c8a02dd
Jie Fu
claimed_verified
2024-10-01T07:16:54.588Z
641a6895fb5ffff5ac79d593
https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
Jie Fu
user
bigaidream
66fa19f693b99e4c7c8a02de
Wenhao Huang
claimed_verified
2024-10-13T20:30:10.239Z
65bb11cb00a03997849e9e85
/avatars/17022b0254192a837f4fe00d84389cda.svg
Wenhao Huang
user
StephenHuang
MIO: A Foundation Model on Multimodal Tokens
In this paper, we introduce MIO, a novel foundation model built on multimodal
tokens, capable of understanding and generating speech, text, images, and
videos in an end-to-end, autoregressive manner. While the emergence of large
language models (LLMs) and multimodal large language models (MM-LLMs) propels
advancements in artificial general intelligence through their versatile
capabilities, they still lack true any-to-any understanding and generation.
Recently, the release of GPT-4o has showcased the remarkable potential of
any-to-any LLMs for complex real-world tasks, enabling omnidirectional input
and output across images, speech, and text. However, it is closed-source and
does not support the generation of multimodal interleaved sequences. To address
this gap, we present MIO, which is trained on a mixture of discrete tokens
across four modalities using causal multimodal modeling. MIO undergoes a
four-stage training process: (1) alignment pre-training, (2) interleaved
pre-training, (3) speech-enhanced pre-training, and (4) comprehensive
supervised fine-tuning on diverse textual, visual, and speech tasks. Our
experimental results indicate that MIO exhibits competitive, and in some cases
superior, performance compared to previous dual-modal baselines, any-to-any
model baselines, and even modality-specific baselines. Moreover, MIO
demonstrates advanced capabilities inherent to its any-to-any feature, such as
interleaved video-text generation, chain-of-visual-thought reasoning, visual
guideline generation, instructional image editing, etc.
MIO, a novel foundation model, achieves competitive performance in multimodal tasks through an end-to-end, autoregressive approach using causal multimodal modeling.
MIO multimodal tokens end-to-end autoregressive large language models multimodal large language models GPT-4o causal multimodal modeling alignment pre-training interleaved pre-training speech-enhanced pre-training comprehensive supervised fine-tuning interleaved video-text generation chain-of-visual-thought reasoning visual guideline generation instructional image imaging