video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models
Speech understanding as an element of the more generic video understanding
using audio-visual large language models (av-LLMs) is a crucial yet
understudied aspect. This paper proposes video-SALMONN, a single end-to-end
av-LLM for video processing, which can understand not only visual frame
sequences, audio events and music, but speech as well. To obtain fine-grained
temporal information required by speech understanding, while keeping efficient
for other video elements, this paper proposes a novel multi-resolution causal
Q-Former (MRC Q-Former) structure to connect pre-trained audio-visual encoders
and the backbone large language model. Moreover, dedicated training approaches
including the diversity loss and the unpaired audio-visual mixed training
scheme are proposed to avoid frames or modality dominance. On the introduced
speech-audio-visual evaluation benchmark, video-SALMONN achieves more than 25\%
absolute accuracy improvements on the video-QA task and over 30\% absolute
accuracy improvements on audio-visual QA tasks with human speech. In addition,
video-SALMONN demonstrates remarkable video comprehension and reasoning
abilities on tasks that are unprecedented by other av-LLMs. Our training code
and model checkpoints are available at
\url{https://github.com/bytedance/SALMONN/}.
667a64f34af5328a70faac49
Guangzhi Sun
admin_assigned
2024-06-25T14:02:41.604Z
64ab544489aa67e4a2505eeb
/avatars/f1a9def3afbec2f8b89ef4450770d67e.svg
Guangzhi Sun
user
BrianatCambridge
667a64f34af5328a70faac4a
Wenyi Yu
667a64f34af5328a70faac4b
Changli Tang
admin_assigned
2024-06-25T14:02:55.818Z
63770389cdcc1bf630870758
/avatars/6bc7c602e79688bc42a4c79ecf5b6d2d.svg
Changli Tang
user
Changli
667a64f34af5328a70faac4c
Xianzhao Chen
admin_assigned
2024-06-25T14:03:01.502Z
65a895f09fa2a0f9a2a5800f
/avatars/a0bc08fec550ff022a1d948d3ad9d42c.svg
xianzhao chen
user
cxzcxzcxz
667a64f34af5328a70faac4d
Tian Tan
667a64f34af5328a70faac4e
Wei Li
667a64f34af5328a70faac4f
Lu Lu
667a64f34af5328a70faac50
Zejun Ma
667a64f34af5328a70faac51
Yuxuan Wang
667a64f34af5328a70faac52
Chao Zhang
video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models
Speech understanding as an element of the more generic video understanding
using audio-visual large language models (av-LLMs) is a crucial yet
understudied aspect. This paper proposes video-SALMONN, a single end-to-end
av-LLM for video processing, which can understand not only visual frame
sequences, audio events and music, but speech as well. To obtain fine-grained
temporal information required by speech understanding, while keeping efficient
for other video elements, this paper proposes a novel multi-resolution causal
Q-Former (MRC Q-Former) structure to connect pre-trained audio-visual encoders
and the backbone large language model. Moreover, dedicated training approaches
including the diversity loss and the unpaired audio-visual mixed training
scheme are proposed to avoid frames or modality dominance. On the introduced
speech-audio-visual evaluation benchmark, video-SALMONN achieves more than 25\%
absolute accuracy improvements on the video-QA task and over 30\% absolute
accuracy improvements on audio-visual QA tasks with human speech. In addition,
video-SALMONN demonstrates remarkable video comprehension and reasoning
abilities on tasks that are unprecedented by other av-LLMs. Our training code
and model checkpoints are available at
\url{https://github.com/bytedance/SALMONN/}.
Video-SALMONN, an end-to-end audio-visual large language model, achieves significant accuracy improvements in speech and audio-visual question answering through a multi-resolution causal Q-Former and dedicated training methods.
audio-visual large language models av-LLMs video-SALMONN visual frame sequences audio events music speech understanding multi-resolution causal Q-Former diversity loss unpaired audio-visual mixed training video-QA task audio-visual QA tasks video comprehension reasoning abilities