OMCAT: Omni Context Aware Transformer
Large Language Models (LLMs) have made significant strides in text generation
and comprehension, with recent advancements extending into multimodal LLMs that
integrate visual and audio inputs. However, these models continue to struggle
with fine-grained, cross-modal temporal understanding, particularly when
correlating events across audio and video streams. We address these challenges
with two key contributions: a new dataset and model, called OCTAV and OMCAT
respectively. OCTAV (Omni Context and Temporal Audio Video) is a novel dataset
designed to capture event transitions across audio and video. Second, OMCAT
(Omni Context Aware Transformer) is a powerful model that leverages RoTE
(Rotary Time Embeddings), an innovative extension of RoPE, to enhance temporal
grounding and computational efficiency in time-anchored tasks. Through a robust
three-stage training pipeline-feature alignment, instruction tuning, and
OCTAV-specific training-OMCAT excels in cross-modal temporal understanding. Our
model demonstrates state-of-the-art performance on Audio-Visual Question
Answering (AVQA) tasks and the OCTAV benchmark, showcasing significant gains in
temporal reasoning and cross-modal alignment, as validated through
comprehensive experiments and ablation studies. Our dataset and code will be
made publicly available. The link to our demo page is https://om-cat.github.io.
67107aea2b203d5fcbbe5b5c
Arushi Goel
extracted_pending
2024-10-17T02:48:12.423Z
65c0b5babf441a1953586df7
/avatars/b767ab08dc096e0aeac29acfcb26d39e.svg
Arushi Goel
user
goarushi27
67107aea2b203d5fcbbe5b5d
Karan Sapra
67107aea2b203d5fcbbe5b5e
Matthieu Le
67107aea2b203d5fcbbe5b5f
Rafael Valle
extracted_pending
2025-02-09T17:18:54.013Z
67a8e3610836885c44bd2ab2
/avatars/ae8d578a47de3302f07430924db3f105.svg
Rafael Valle
user
rafaelvallenvidia
67107aea2b203d5fcbbe5b60
Andrew Tao
67107aea2b203d5fcbbe5b61
Bryan Catanzaro
OMCAT: Omni Context Aware Transformer
Large Language Models (LLMs) have made significant strides in text generation
and comprehension, with recent advancements extending into multimodal LLMs that
integrate visual and audio inputs. However, these models continue to struggle
with fine-grained, cross-modal temporal understanding, particularly when
correlating events across audio and video streams. We address these challenges
with two key contributions: a new dataset and model, called OCTAV and OMCAT
respectively. OCTAV (Omni Context and Temporal Audio Video) is a novel dataset
designed to capture event transitions across audio and video. Second, OMCAT
(Omni Context Aware Transformer) is a powerful model that leverages RoTE
(Rotary Time Embeddings), an innovative extension of RoPE, to enhance temporal
grounding and computational efficiency in time-anchored tasks. Through a robust
three-stage training pipeline-feature alignment, instruction tuning, and
OCTAV-specific training-OMCAT excels in cross-modal temporal understanding. Our
model demonstrates state-of-the-art performance on Audio-Visual Question
Answering (AVQA) tasks and the OCTAV benchmark, showcasing significant gains in
temporal reasoning and cross-modal alignment, as validated through
comprehensive experiments and ablation studies. Our dataset and code will be
made publicly available. The link to our demo page is https://om-cat.github.io.
New model and dataset, OMCAT and OCTAV, improve cross-modal temporal understanding by leveraging Rotary Time Embeddings for audio-visual question answering tasks.
Large Language Models multimodal LLMs fine-grained cross-modal temporal understanding OCTAV OMCAT Omni Context and Temporal Audio Video Omni Context Aware Transformer Rotary Time Embeddings RoTE RoPE temporal grounding three-stage training pipeline feature alignment instruction tuning OCTAV-specific training Audio-Visual Question Answering (AVQA) temporal reasoning cross-modal alignment