InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn
  Dialogue
We introduce InteractiveOmni, a unified and open-source omni-modal large
language model for audio-visual multi-turn interaction, ranging from 4B to 8B
parameters, designed to lead the field of lightweight models by offering
comprehensive omni-modal understanding and speech generation capabilities. To
achieve this, we integrate the vision encoder, audio encoder, large language
model, and speech decoder into a unified model for understanding and generation
tasks. We design a multi-stage training strategy to ensure robust cross-modal
capabilities, including pre-training for omni-modal understanding, followed by
post-training with speech conversation and audio-visual interaction. To enable
human-like long-term conversational ability, we meticulously curate a
multi-turn training dataset that enhances the model's ability to handle complex
and multi-turn interactions. To effectively evaluate the multi-turn memory and
speech interaction capabilities, we construct the multi-modal multi-turn memory
benchmark and the multi-turn speech interaction benchmark. Experiments
demonstrate that InteractiveOmni significantly outperforms leading open-source
models and provides a more intelligent multi-turn audio-visual experience,
particularly in its long-term memory capabilities. Notably, InteractiveOmni-4B
is comparable to the much larger model like Qwen2.5-Omni-7B on general
benchmarks, and it can retain 97% of the performance of the InteractiveOmni-8B
while utilizing only 50% of the model size. Achieving state-of-the-art results
against similarly sized models across image, audio, video understanding, and
speech generation tasks, InteractiveOmni is an accessible, open-source
foundation for next-generation intelligent interactive systems.
68f0571d36f8b025381e1957
Wenwen Tong
claimed_verified
2025-10-17T04:14:03.126Z
647d4f1236e109abce409c3b
/avatars/d166f5f8be666e96b522a0a0effd21c4.svg
Wenwen Tong
user
tongww
68f0571d36f8b025381e1958
Hewei Guo
claimed_verified
2025-10-16T06:38:36.074Z
662f58bc5cf9dca4a2b225ca
/avatars/4b4f989621ddf2882b3dcdc43de4daf0.svg
Hewei Guo
user
heweiguo
68f0571d36f8b025381e1959
Dongchuan Ran
68f0571d36f8b025381e195a
Jiangnan Chen
claimed_verified
2025-10-16T07:25:38.654Z
65531be65d55ccb20e97f34a
/avatars/ba919227d8c0b3532b62bb3a2d2d6ff3.svg
Chan
user
Chris2me
68f0571d36f8b025381e195b
Jiefan Lu
68f0571d36f8b025381e195c
Kaibin Wang
68f0571d36f8b025381e195d
Keqiang Li
68f0571d36f8b025381e195e
Xiaoxu Zhu
68f0571d36f8b025381e195f
Jiakui Li
68f0571d36f8b025381e1960
Kehan Li
68f0571d36f8b025381e1961
Xueheng Li
68f0571d36f8b025381e1962
Lumin Li
68f0571d36f8b025381e1963
Chenxu Guo
68f0571d36f8b025381e1964
Jiasheng Zhou
68f0571d36f8b025381e1965
Jiandong Chen
68f0571d36f8b025381e1966
Xianye Wu
68f0571d36f8b025381e1967
Jiahao Wang
claimed_verified
2025-10-16T11:40:32.492Z
66bc7862aa7cdcb1c31a1efb
/avatars/4c2ab907247fe071ff5cdd71c404ca7c.svg
wang jiahao
user
datamonkey
68f0571d36f8b025381e1968
Silei Wu
68f0571d36f8b025381e1969
Lei Chen
68f0571d36f8b025381e196a
Hanming Deng
68f0571d36f8b025381e196b
Yuxuan Song
68f0571d36f8b025381e196c
Dinghao Zhou
68f0571d36f8b025381e196d
Guiping Zhong
68f0571d36f8b025381e196e
Ken Zheng
68f0571d36f8b025381e196f
Shiyin Kang
68f0571d36f8b025381e1970
Lewei Lu
InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn
  Dialogue
We introduce InteractiveOmni, a unified and open-source omni-modal large
language model for audio-visual multi-turn interaction, ranging from 4B to 8B
parameters, designed to lead the field of lightweight models by offering
comprehensive omni-modal understanding and speech generation capabilities. To
achieve this, we integrate the vision encoder, audio encoder, large language
model, and speech decoder into a unified model for understanding and generation
tasks. We design a multi-stage training strategy to ensure robust cross-modal
capabilities, including pre-training for omni-modal understanding, followed by
post-training with speech conversation and audio-visual interaction. To enable
human-like long-term conversational ability, we meticulously curate a
multi-turn training dataset that enhances the model's ability to handle complex
and multi-turn interactions. To effectively evaluate the multi-turn memory and
speech interaction capabilities, we construct the multi-modal multi-turn memory
benchmark and the multi-turn speech interaction benchmark. Experiments
demonstrate that InteractiveOmni significantly outperforms leading open-source
models and provides a more intelligent multi-turn audio-visual experience,
particularly in its long-term memory capabilities. Notably, InteractiveOmni-4B
is comparable to the much larger model like Qwen2.5-Omni-7B on general
benchmarks, and it can retain 97% of the performance of the InteractiveOmni-8B
while utilizing only 50% of the model size. Achieving state-of-the-art results
against similarly sized models across image, audio, video understanding, and
speech generation tasks, InteractiveOmni is an accessible, open-source
foundation for next-generation intelligent interactive systems.
InteractiveOmni is a unified omni-modal large language model for audio-visual multi-turn interactions, offering comprehensive understanding and speech generation capabilities with efficient parameter usage.
omni-modal large language model audio-visual multi-turn interaction vision encoder audio encoder large language model speech decoder multi-stage training strategy omni-modal understanding speech conversation audio-visual interaction multi-turn training dataset multi-modal multi-turn memory benchmark multi-turn speech interaction benchmark long-term memory capabilities Qwen2.5-Omni-7B InteractiveOmni-4B InteractiveOmni-8B image understanding audio understanding video understanding speech generation