LITA: Language Instructed Temporal-Localization Assistant
There has been tremendous progress in multimodal Large Language Models
(LLMs). Recent works have extended these models to video input with promising
instruction following capabilities. However, an important missing piece is
temporal localization. These models cannot accurately answer the "When?"
questions. We identify three key aspects that limit their temporal localization
capabilities: (i) time representation, (ii) architecture, and (iii) data. We
address these shortcomings by proposing Language Instructed
Temporal-Localization Assistant (LITA) with the following features: (1) We
introduce time tokens that encode timestamps relative to the video length to
better represent time in videos. (2) We introduce SlowFast tokens in the
architecture to capture temporal information at fine temporal resolution. (3)
We emphasize temporal localization data for LITA. In addition to leveraging
existing video datasets with timestamps, we propose a new task, Reasoning
Temporal Localization (RTL), along with the dataset, ActivityNet-RTL, for
learning and evaluating this task. Reasoning temporal localization requires
both the reasoning and temporal localization of Video LLMs. LITA demonstrates
strong performance on this challenging task, nearly doubling the temporal mean
intersection-over-union (mIoU) of baselines. In addition, we show that our
emphasis on temporal localization also substantially improves video-based text
generation compared to existing Video LLMs, including a 36% relative
improvement of Temporal Understanding. Code is available at:
https://github.com/NVlabs/LITA
660645469f5a7bd168fa41df
De-An Huang
660645469f5a7bd168fa41e0
Shijia Liao
660645469f5a7bd168fa41e1
Subhashree Radhakrishnan
660645469f5a7bd168fa41e2
Hongxu Yin
claimed_verified
2025-03-27T09:06:20.229Z
65a8b7f69aec1645994e7a15
/avatars/debc086f3fea029db22847bde80799a0.svg
Hongxu Yin
user
yinhongxu
660645469f5a7bd168fa41e3
Pavlo Molchanov
claimed_verified
2024-07-24T07:29:25.549Z
646d0c1c534e52f8c30500a6
https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
Pavlo Molchanov
user
pmolchanov
660645469f5a7bd168fa41e4
Zhiding Yu
claimed_verified
2024-08-29T07:25:15.228Z
66c8037c737ba92ae3fe0322
https://cdn-avatars.huggingface.co/v1/production/uploads/66c8037c737ba92ae3fe0322/WR_Yh5DWOVVh7IFlF24NM.jpeg
Zhiding Yu
user
Zhiding
660645469f5a7bd168fa41e5
Jan Kautz
LITA: Language Instructed Temporal-Localization Assistant
There has been tremendous progress in multimodal Large Language Models
(LLMs). Recent works have extended these models to video input with promising
instruction following capabilities. However, an important missing piece is
temporal localization. These models cannot accurately answer the "When?"
questions. We identify three key aspects that limit their temporal localization
capabilities: (i) time representation, (ii) architecture, and (iii) data. We
address these shortcomings by proposing Language Instructed
Temporal-Localization Assistant (LITA) with the following features: (1) We
introduce time tokens that encode timestamps relative to the video length to
better represent time in videos. (2) We introduce SlowFast tokens in the
architecture to capture temporal information at fine temporal resolution. (3)
We emphasize temporal localization data for LITA. In addition to leveraging
existing video datasets with timestamps, we propose a new task, Reasoning
Temporal Localization (RTL), along with the dataset, ActivityNet-RTL, for
learning and evaluating this task. Reasoning temporal localization requires
both the reasoning and temporal localization of Video LLMs. LITA demonstrates
strong performance on this challenging task, nearly doubling the temporal mean
intersection-over-union (mIoU) of baselines. In addition, we show that our
emphasis on temporal localization also substantially improves video-based text
generation compared to existing Video LLMs, including a 36% relative
improvement of Temporal Understanding. Code is available at:
https://github.com/NVlabs/LITA
LITA addresses temporal localization in multimodal LLMs by introducing time tokens, SlowFast architecture, and a new task dataset, improving video-based text generation and temporal mIoU.
multimodal LLMs time representation architecture data time tokens SlowFast tokens temporal localization Reasoning Temporal Localization (RTL) ActivityNet-RTL temporal mean intersection-over-union (mIoU) Temporal Understanding