Tarsier2: Advancing Large Vision-Language Models from Detailed Video
  Description to Comprehensive Video Understanding
We introduce Tarsier2, a state-of-the-art large vision-language model (LVLM)
designed for generating detailed and accurate video descriptions, while also
exhibiting superior general video understanding capabilities. Tarsier2 achieves
significant advancements through three key upgrades: (1) Scaling pre-training
data from 11M to 40M video-text pairs, enriching both volume and diversity; (2)
Performing fine-grained temporal alignment during supervised fine-tuning; (3)
Using model-based sampling to automatically construct preference data and
applying DPO training for optimization. Extensive experiments show that
Tarsier2-7B consistently outperforms leading proprietary models, including
GPT-4o and Gemini 1.5 Pro, in detailed video description tasks. On the DREAM-1K
benchmark, Tarsier2-7B improves F1 by 2.8\% over GPT-4o and 5.8\% over
Gemini-1.5-Pro. In human side-by-side evaluations, Tarsier2-7B shows a +8.6\%
performance advantage over GPT-4o and +24.9\% over Gemini-1.5-Pro. Tarsier2-7B
also sets new state-of-the-art results across 15 public benchmarks, spanning
tasks such as video question-answering, video grounding, hallucination test,
and embodied question-answering, demonstrating its versatility as a robust
generalist vision-language model.
67873cdd1f65db189f9f64d7
Liping Yuan
admin_assigned
2025-01-15T17:07:12.581Z
6638cd255b81e56d337a4a98
/avatars/b00805982755e4838b5ce0b23e3357c2.svg
Liping Yuan
user
yuanlp
67873cdd1f65db189f9f64d8
Jiawei Wang
claimed_verified
2025-01-15T21:21:17.147Z
6788122301d5ba1d3eff23ba
/avatars/203dc8e1d542be55ea16eafcd7f396ff.svg
Jiawei Wang
user
0nejiawei
67873cdd1f65db189f9f64d9
Haomiao Sun
admin_assigned
2025-01-15T17:07:46.243Z
668685161589885791b76c25
/avatars/ffb9339114b75205ab5840b70f59f4e5.svg
sunhaomiao
user
sunhm15
67873cdd1f65db189f9f64da
Yuchen Zhang
67873cdd1f65db189f9f64db
Yuan Lin
Tarsier2: Advancing Large Vision-Language Models from Detailed Video
  Description to Comprehensive Video Understanding
We introduce Tarsier2, a state-of-the-art large vision-language model (LVLM)
designed for generating detailed and accurate video descriptions, while also
exhibiting superior general video understanding capabilities. Tarsier2 achieves
significant advancements through three key upgrades: (1) Scaling pre-training
data from 11M to 40M video-text pairs, enriching both volume and diversity; (2)
Performing fine-grained temporal alignment during supervised fine-tuning; (3)
Using model-based sampling to automatically construct preference data and
applying DPO training for optimization. Extensive experiments show that
Tarsier2-7B consistently outperforms leading proprietary models, including
GPT-4o and Gemini 1.5 Pro, in detailed video description tasks. On the DREAM-1K
benchmark, Tarsier2-7B improves F1 by 2.8\% over GPT-4o and 5.8\% over
Gemini-1.5-Pro. In human side-by-side evaluations, Tarsier2-7B shows a +8.6\%
performance advantage over GPT-4o and +24.9\% over Gemini-1.5-Pro. Tarsier2-7B
also sets new state-of-the-art results across 15 public benchmarks, spanning
tasks such as video question-answering, video grounding, hallucination test,
and embodied question-answering, demonstrating its versatility as a robust
generalist vision-language model.
Tarsier2, a large vision-language model, achieves state-of-the-art performance in video description and understanding by scaling pre-training data, fine-grained temporal alignment, and using DPO training.
large vision-language model video description fine-grained temporal alignment model-based sampling DPO training video question-answering video grounding hallucination test embodied question-answering