Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation
In recent years, there has been a significant surge of interest in unifying
image comprehension and generation within Large Language Models (LLMs). This
growing interest has prompted us to explore extending this unification to
videos. The core challenge lies in developing a versatile video tokenizer that
captures both the spatial characteristics and temporal dynamics of videos to
obtain representations for LLMs, and the representations can be further decoded
into realistic video clips to enable video generation. In this work, we
introduce Divot, a Diffusion-Powered Video Tokenizer, which leverages the
diffusion process for self-supervised video representation learning. We posit
that if a video diffusion model can effectively de-noise video clips by taking
the features of a video tokenizer as the condition, then the tokenizer has
successfully captured robust spatial and temporal information. Additionally,
the video diffusion model inherently functions as a de-tokenizer, decoding
videos from their representations. Building upon the Divot tokenizer, we
present Divot-Vicuna through video-to-text autoregression and text-to-video
generation by modeling the distributions of continuous-valued Divot features
with a Gaussian Mixture Model. Experimental results demonstrate that our
diffusion-based video tokenizer, when integrated with a pre-trained LLM,
achieves competitive performance across various video comprehension and
generation benchmarks. The instruction tuned Divot-Vicuna also excels in video
storytelling, generating interleaved narratives and corresponding videos.
675260813c0a3886978b2e48
Yuying Ge
admin_assigned
2024-12-10T16:16:42.443Z
6455cc8f654d8bccae50e4d4
/avatars/506a9992e5bf52e06d37cc22e4b307c0.svg
Yuying Ge
user
tttoaster
675260813c0a3886978b2e49
Yizhuo Li
admin_assigned
2024-12-10T16:16:48.137Z
630b094f8b327c7b8b94d24c
https://cdn-avatars.huggingface.co/v1/production/uploads/630b094f8b327c7b8b94d24c/Fd0ugLOHJZIi8oUZScOmX.jpeg
Yizhuo Li
user
liyz
675260813c0a3886978b2e4a
Yixiao Ge
admin_assigned
2024-12-10T16:16:54.425Z
640e9762b03f4cd29f58d982
/avatars/81da37d628163fe3e094b247c7c3a3b5.svg
Yixiao Ge
user
yxgeee
675260813c0a3886978b2e4b
Ying Shan
admin_assigned
2024-12-10T16:17:01.613Z
63ca3ddc04c979828310bfcb
/avatars/615e0d8622950b4408b40d550f02a894.svg
Ying Shan
user
yshan2u
Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation
In recent years, there has been a significant surge of interest in unifying
image comprehension and generation within Large Language Models (LLMs). This
growing interest has prompted us to explore extending this unification to
videos. The core challenge lies in developing a versatile video tokenizer that
captures both the spatial characteristics and temporal dynamics of videos to
obtain representations for LLMs, and the representations can be further decoded
into realistic video clips to enable video generation. In this work, we
introduce Divot, a Diffusion-Powered Video Tokenizer, which leverages the
diffusion process for self-supervised video representation learning. We posit
that if a video diffusion model can effectively de-noise video clips by taking
the features of a video tokenizer as the condition, then the tokenizer has
successfully captured robust spatial and temporal information. Additionally,
the video diffusion model inherently functions as a de-tokenizer, decoding
videos from their representations. Building upon the Divot tokenizer, we
present Divot-Vicuna through video-to-text autoregression and text-to-video
generation by modeling the distributions of continuous-valued Divot features
with a Gaussian Mixture Model. Experimental results demonstrate that our
diffusion-based video tokenizer, when integrated with a pre-trained LLM,
achieves competitive performance across various video comprehension and
generation benchmarks. The instruction tuned Divot-Vicuna also excels in video
storytelling, generating interleaved narratives and corresponding videos.
A diffusion-powered video tokenizer (Divot) captures spatial and temporal video features for integration with LLMs, enabling video comprehension and generation with competitive performance.
diffusion-Powered Video Tokenizer diffusion process self-supervised video representation learning video diffusion model Gaussian Mixture Model video-to-text autoregression text-to-video generation video storytelling