Vidi: Large Multimodal Models for Video Understanding and Editing
Humans naturally share information with those they are connected to, and
video has become one of the dominant mediums for communication and expression
on the Internet. To support the creation of high-quality large-scale video
content, a modern pipeline requires a comprehensive understanding of both the
raw input materials (e.g., the unedited footage captured by cameras) and the
editing components (e.g., visual effects). In video editing scenarios, models
must process multiple modalities (e.g., vision, audio, text) with strong
background knowledge and handle flexible input lengths (e.g., hour-long raw
videos), which poses significant challenges for traditional models. In this
report, we introduce Vidi, a family of Large Multimodal Models (LMMs) for a
wide range of video understand editing scenarios. The first release focuses on
temporal retrieval, i.e., identifying the time ranges within the input videos
corresponding to a given text query, which plays a critical role in intelligent
editing. The model is capable of processing hour-long videos with strong
temporal understanding capability, e.g., retrieve time ranges for certain
queries. To support a comprehensive evaluation in real-world scenarios, we also
present the VUE-TR benchmark, which introduces five key advancements. 1) Video
duration: significantly longer than existing temporal retrival datasets, 2)
Audio support: includes audio-based queries, 3) Query format: diverse query
lengths/formats, 4) Annotation quality: ground-truth time ranges are manually
annotated. 5) Evaluation metric: a refined IoU metric to support evaluation
over multiple time ranges. Remarkably, Vidi significantly outperforms leading
proprietary models, e.g., GPT-4o and Gemini, on the temporal retrieval task,
indicating its superiority in video editing scenarios.
680846defa5a6cc6bd9d2cf3
Vidi Team
680846defa5a6cc6bd9d2cf4
Celong Liu
admin_assigned
2025-04-23T13:52:25.362Z
64364ba31adb261e94e0e5b4
/avatars/cb6ef2155d88b1e64c4ff36bd3a8f255.svg
Celong Liu
user
goddice
680846defa5a6cc6bd9d2cf5
Chia-Wen Kuo
admin_assigned
2025-04-23T13:52:32.272Z
64eea9ff9884f64fd27544f5
/avatars/65aac158a55f16c82605f4dafd59b3ea.svg
Chia-Wen Kuo
user
cwkuo
680846defa5a6cc6bd9d2cf6
Dawei Du
claimed_verified
2025-04-23T08:28:04.152Z
6476af4402fc644c810b29a2
/avatars/68aefabe6b000443f4601137e6672187.svg
Dawei Du
user
daviddousa
680846defa5a6cc6bd9d2cf7
Fan Chen
claimed_verified
2025-05-06T08:34:33.789Z
65c2a2f5b911dda8cb3ff13b
/avatars/29ad2ed1433be78e1f1ceefbe7463a4e.svg
Fan Chen
user
fanchenBYT
680846defa5a6cc6bd9d2cf8
Guang Chen
680846defa5a6cc6bd9d2cf9
Jiamin Yuan
680846defa5a6cc6bd9d2cfa
Lingxi Zhang
680846defa5a6cc6bd9d2cfb
Lu Guo
680846defa5a6cc6bd9d2cfc
Lusha Li
680846defa5a6cc6bd9d2cfd
Longyin Wen
admin_assigned
2025-04-23T13:53:13.545Z
644df7eacfb40c94eae71186
/avatars/1daa4967efd34d54c59aa95970093dbd.svg
Longyin Wen
user
lionwen
680846defa5a6cc6bd9d2cfe
Qingyu Chen
680846defa5a6cc6bd9d2cff
Rachel Deng
admin_assigned
2025-04-23T13:53:24.615Z
6509f43b1aece923f200ba50
/avatars/d6f7c96222ae8442291d193a1aa768c3.svg
Rachel Deng
user
racheldeng
680846defa5a6cc6bd9d2d00
Sijie Zhu
admin_assigned
2025-04-23T13:53:31.960Z
642ee3316b75297bdcccc17e
/avatars/2ca159e38c64b203835cd0086cf5abb1.svg
zhusijie
user
sijiezhu
680846defa5a6cc6bd9d2d01
Stuart Siew
680846defa5a6cc6bd9d2d02
Tong Jin
680846defa5a6cc6bd9d2d03
Wei Lu
680846defa5a6cc6bd9d2d04
Wen Zhong
680846defa5a6cc6bd9d2d05
Xiaohui Shen
admin_assigned
2025-04-23T13:53:43.973Z
6430aa1b32a732121cd81f98
/avatars/5419f8d6d4d36fa5ac83e30667b9fd99.svg
Xiaohui Shen
user
XiaohuiShen
680846defa5a6cc6bd9d2d06
Xin Gu
680846defa5a6cc6bd9d2d07
Xing Mei
680846defa5a6cc6bd9d2d08
Xueqiong Qu
Vidi: Large Multimodal Models for Video Understanding and Editing
Humans naturally share information with those they are connected to, and
video has become one of the dominant mediums for communication and expression
on the Internet. To support the creation of high-quality large-scale video
content, a modern pipeline requires a comprehensive understanding of both the
raw input materials (e.g., the unedited footage captured by cameras) and the
editing components (e.g., visual effects). In video editing scenarios, models
must process multiple modalities (e.g., vision, audio, text) with strong
background knowledge and handle flexible input lengths (e.g., hour-long raw
videos), which poses significant challenges for traditional models. In this
report, we introduce Vidi, a family of Large Multimodal Models (LMMs) for a
wide range of video understand editing scenarios. The first release focuses on
temporal retrieval, i.e., identifying the time ranges within the input videos
corresponding to a given text query, which plays a critical role in intelligent
editing. The model is capable of processing hour-long videos with strong
temporal understanding capability, e.g., retrieve time ranges for certain
queries. To support a comprehensive evaluation in real-world scenarios, we also
present the VUE-TR benchmark, which introduces five key advancements. 1) Video
duration: significantly longer than existing temporal retrival datasets, 2)
Audio support: includes audio-based queries, 3) Query format: diverse query
lengths/formats, 4) Annotation quality: ground-truth time ranges are manually
annotated. 5) Evaluation metric: a refined IoU metric to support evaluation
over multiple time ranges. Remarkably, Vidi significantly outperforms leading
proprietary models, e.g., GPT-4o and Gemini, on the temporal retrieval task,
indicating its superiority in video editing scenarios.
Vidi, a family of Large Multimodal Models, excels in temporal retrieval for video editing by processing long, multimodal video content and outperforming proprietary models on the VUE-TR benchmark.
LMMs temporal retrieval multimodal models video editing temporal understanding VUE-TR benchmark