xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video
  Even in VLMs
We present xGen-MM-Vid (BLIP-3-Video): a multimodal language model for
videos, particularly designed to efficiently capture temporal information over
multiple frames. BLIP-3-Video takes advantage of the 'temporal encoder' in
addition to the conventional visual tokenizer, which maps a sequence of tokens
over multiple frames into a compact set of visual tokens. This enables
BLIP3-Video to use much fewer visual tokens than its competing models (e.g., 32
vs. 4608 tokens). We explore different types of temporal encoders, including
learnable spatio-temporal pooling as well as sequential models like Token
Turing Machines. We experimentally confirm that BLIP-3-Video obtains video
question-answering accuracies comparable to much larger state-of-the-art models
(e.g., 34B), while being much smaller (i.e., 4B) and more efficient by using
fewer visual tokens. The project website is at
https://www.salesforceairesearch.com/opensource/xGen-MM-Vid/index.html
6718177310ca4f9b8f0be3f8
Michael S. Ryoo
extracted_pending
2024-10-22T21:21:57.508Z
670d93425c92b289329b19db
/avatars/c7c9b67c331f8974384780e9ce1958a2.svg
Michael Ryoo
user
michaelryoo
6718177310ca4f9b8f0be3f9
Honglu Zhou
6718177310ca4f9b8f0be3fa
Shrikant Kendre
6718177310ca4f9b8f0be3fb
Can Qin
6718177310ca4f9b8f0be3fc
Le Xue
6718177310ca4f9b8f0be3fd
Manli Shu
6718177310ca4f9b8f0be3fe
Silvio Savarese
6718177310ca4f9b8f0be3ff
Ran Xu
claimed_verified
2025-05-15T10:41:26.876Z
6465c4c863e7e09dd02e3e1b
/avatars/200b029184d2616f98296a2c212f0785.svg
Ran Xu
user
xurantju
6718177310ca4f9b8f0be400
Caiming Xiong
6718177310ca4f9b8f0be401
Juan Carlos Niebles
xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video
  Even in VLMs
We present xGen-MM-Vid (BLIP-3-Video): a multimodal language model for
videos, particularly designed to efficiently capture temporal information over
multiple frames. BLIP-3-Video takes advantage of the 'temporal encoder' in
addition to the conventional visual tokenizer, which maps a sequence of tokens
over multiple frames into a compact set of visual tokens. This enables
BLIP3-Video to use much fewer visual tokens than its competing models (e.g., 32
vs. 4608 tokens). We explore different types of temporal encoders, including
learnable spatio-temporal pooling as well as sequential models like Token
Turing Machines. We experimentally confirm that BLIP-3-Video obtains video
question-answering accuracies comparable to much larger state-of-the-art models
(e.g., 34B), while being much smaller (i.e., 4B) and more efficient by using
fewer visual tokens. The project website is at
https://www.salesforceairesearch.com/opensource/xGen-MM-Vid/index.html
BLIP-3-Video is a video-specific multimodal language model using fewer visual tokens and a temporal encoder for efficient temporal information capture and accurate video question-answering.
multimodal language model temporal information temporal encoder visual tokenizer spatio-temporal pooling Token Turing Machines video question-answering visual tokens