xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video
  Even in VLMs
We present xGen-MM-Vid (BLIP-3-Video): a multimodal language model for
videos, particularly designed to efficiently capture temporal information over
multiple frames. BLIP-3-Video takes advantage of the 'temporal encoder' in
addition to the conventional visual tokenizer, which maps a sequence of tokens
over multiple frames into a compact set of visual tokens. This enables
BLIP3-Video to use much fewer visual tokens than its competing models (e.g., 32
vs. 4608 tokens). We explore different types of temporal encoders, including
learnable spatio-temporal pooling as well as sequential models like Token
Turing Machines. We experimentally confirm that BLIP-3-Video obtains video
question-answering accuracies comparable to much larger state-of-the-art models
(e.g., 34B), while being much smaller (i.e., 4B) and more efficient by using
fewer visual tokens. The project website is at
https://www.salesforceairesearch.com/opensource/xGen-MM-Vid/index.html
6718177310ca4f9b8f0be3f8
Michael S. Ryoo
extracted_pending
2024-10-22T21:21:57.508Z
670d93425c92b289329b19db
/avatars/c7c9b67c331f8974384780e9ce1958a2.svg
Michael Ryoo
user
michaelryoo
6718177310ca4f9b8f0be3f9
Honglu Zhou
6718177310ca4f9b8f0be3fa
Shrikant Kendre
6718177310ca4f9b8f0be3fb
Can Qin
6718177310ca4f9b8f0be3fc
Le Xue
6718177310ca4f9b8f0be3fd
Manli Shu
6718177310ca4f9b8f0be3fe
Silvio Savarese
6718177310ca4f9b8f0be3ff
Ran Xu
6718177310ca4f9b8f0be400
Caiming Xiong
6718177310ca4f9b8f0be401
Juan Carlos Niebles
xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video
  Even in VLMs
We present xGen-MM-Vid (BLIP-3-Video): a multimodal language model for
videos, particularly designed to efficiently capture temporal information over
multiple frames. BLIP-3-Video takes advantage of the 'temporal encoder' in
addition to the conventional visual tokenizer, which maps a sequence of tokens
over multiple frames into a compact set of visual tokens. This enables
BLIP3-Video to use much fewer visual tokens than its competing models (e.g., 32
vs. 4608 tokens). We explore different types of temporal encoders, including
learnable spatio-temporal pooling as well as sequential models like Token
Turing Machines. We experimentally confirm that BLIP-3-Video obtains video
question-answering accuracies comparable to much larger state-of-the-art models
(e.g., 34B), while being much smaller (i.e., 4B) and more efficient by using
fewer visual tokens. The project website is at
https://www.salesforceairesearch.com/opensource/xGen-MM-Vid/index.html
multimodal language model temporal encoder visual tokenizer token Turing Machines video question-answering