OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding
  LLM
Advancing machine intelligence requires developing the ability to perceive
across multiple modalities, much as humans sense the world. We introduce
OmniVinci, an initiative to build a strong, open-source, omni-modal LLM. We
carefully study the design choices across model architecture and data curation.
For model architecture, we present three key innovations: (i) OmniAlignNet for
strengthening alignment between vision and audio embeddings in a shared
omni-modal latent space; (ii) Temporal Embedding Grouping for capturing
relative temporal alignment between vision and audio signals; and (iii)
Constrained Rotary Time Embedding for encoding absolute temporal information in
omni-modal embeddings. We introduce a curation and synthesis pipeline that
generates 24M single-modal and omni-modal conversations. We find that
modalities reinforce one another in both perception and reasoning. Our model,
OmniVinci, outperforms Qwen2.5-Omni with +19.05 on DailyOmni (cross-modal
understanding), +1.7 on MMAR (audio), and +3.9 on Video-MME (vision), while
using just 0.2T training tokens - a 6 times reduction compared to
Qwen2.5-Omni's 1.2T. We finally demonstrate omni-modal advantages in downstream
applications spanning robotics, medical AI, and smart factory.
68f592478589920bf4d32084
Hanrong Ye
68f592478589920bf4d32085
Chao-Han Huck Yang
claimed_verified
2025-10-20T12:11:24.329Z
629e1b71bb6419817ed7566c
https://cdn-avatars.huggingface.co/v1/production/uploads/629e1b71bb6419817ed7566c/0ZCt-11eQtRDCOk9AozOp.jpeg
Huck Yang
user
huckiyang
68f592478589920bf4d32086
Arushi Goel
68f592478589920bf4d32087
Wei Huang
claimed_verified
2025-10-23T15:01:39.826Z
656db3f53dc1d277e5a64410
https://cdn-avatars.huggingface.co/v1/production/uploads/656db3f53dc1d277e5a64410/9kiY2K3MCRcBDk7MrkTBK.png
Wei Huang
user
AaronHuangWei
68f592478589920bf4d32088
Ligeng Zhu
68f592478589920bf4d32089
Yuanhang Su
68f592478589920bf4d3208a
Sean Lin
68f592478589920bf4d3208b
An-Chieh Cheng
68f592478589920bf4d3208c
Zhen Wan
68f592478589920bf4d3208d
Jinchuan Tian
68f592478589920bf4d3208e
Yuming Lou
68f592478589920bf4d3208f
Dong Yang
68f592478589920bf4d32090
Zhijian Liu
68f592478589920bf4d32091
Yukang Chen
68f592478589920bf4d32092
Ambrish Dantrey
68f592478589920bf4d32093
Ehsan Jahangiri
68f592478589920bf4d32094
Sreyan Ghosh
claimed_verified
2025-10-21T14:05:10.019Z
62c9664eb34e600d7eaa4beb
/avatars/ca23ecdec2d31c99ecce97d9b180ae0c.svg
Ghosh
user
Sreyan88
68f592478589920bf4d32095
Daguang Xu
68f592478589920bf4d32096
Ehsan Hosseini-Asl
68f592478589920bf4d32097
Danial Mohseni Taheri
68f592478589920bf4d32098
Vidya Murali
68f592478589920bf4d32099
Sifei Liu
68f592478589920bf4d3209a
Jason Lu
68f592478589920bf4d3209b
Oluwatobi Olabiyi
68f592478589920bf4d3209c
Frank Wang
68f592478589920bf4d3209d
Rafael Valle
68f592478589920bf4d3209e
Bryan Catanzaro
68f592478589920bf4d3209f
Andrew Tao
68f592478589920bf4d320a0
Song Han
68f592478589920bf4d320a1
Jan Kautz
68f592478589920bf4d320a2
Hongxu Yin
68f592478589920bf4d320a3
Pavlo Molchanov
OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding
  LLM
Advancing machine intelligence requires developing the ability to perceive
across multiple modalities, much as humans sense the world. We introduce
OmniVinci, an initiative to build a strong, open-source, omni-modal LLM. We
carefully study the design choices across model architecture and data curation.
For model architecture, we present three key innovations: (i) OmniAlignNet for
strengthening alignment between vision and audio embeddings in a shared
omni-modal latent space; (ii) Temporal Embedding Grouping for capturing
relative temporal alignment between vision and audio signals; and (iii)
Constrained Rotary Time Embedding for encoding absolute temporal information in
omni-modal embeddings. We introduce a curation and synthesis pipeline that
generates 24M single-modal and omni-modal conversations. We find that
modalities reinforce one another in both perception and reasoning. Our model,
OmniVinci, outperforms Qwen2.5-Omni with +19.05 on DailyOmni (cross-modal
understanding), +1.7 on MMAR (audio), and +3.9 on Video-MME (vision), while
using just 0.2T training tokens - a 6 times reduction compared to
Qwen2.5-Omni's 1.2T. We finally demonstrate omni-modal advantages in downstream
applications spanning robotics, medical AI, and smart factory.
OmniVinci, an open-source omni-modal LLM, enhances cross-modal understanding and performance across audio, vision, and robotics applications with innovative architecture and efficient data curation.
OmniAlignNet Temporal Embedding Grouping Constrained Rotary Time Embedding omni-modal latent space DailyOmni MMAR Video-MME omni-modal conversations omni-modal advantages