VidTok: A Versatile and Open-Source Video Tokenizer
Encoding video content into compact latent tokens has become a fundamental
step in video generation and understanding, driven by the need to address the
inherent redundancy in pixel-level representations. Consequently, there is a
growing demand for high-performance, open-source video tokenizers as
video-centric research gains prominence. We introduce VidTok, a versatile video
tokenizer that delivers state-of-the-art performance in both continuous and
discrete tokenizations. VidTok incorporates several key advancements over
existing approaches: 1) model architecture such as convolutional layers and
up/downsampling modules; 2) to address the training instability and codebook
collapse commonly associated with conventional Vector Quantization (VQ), we
integrate Finite Scalar Quantization (FSQ) into discrete video tokenization; 3)
improved training strategies, including a two-stage training process and the
use of reduced frame rates. By integrating these advancements, VidTok achieves
substantial improvements over existing methods, demonstrating superior
performance across multiple metrics, including PSNR, SSIM, LPIPS, and FVD,
under standardized evaluation settings.
6762c51eeceb9f88f41d1c23
Anni Tang
6762c51eeceb9f88f41d1c24
Tianyu He
claimed_verified
2024-12-19T18:01:58.814Z
619b7b1cab4c7b7f16a7d59e
https://cdn-avatars.huggingface.co/v1/production/uploads/619b7b1cab4c7b7f16a7d59e/6TvXaAqBghAMYO1-j5l4v.jpeg
Tianyu He
user
deeptimhe
6762c51eeceb9f88f41d1c25
Junliang Guo
admin_assigned
2024-12-20T10:14:02.510Z
66868659ccb9539da85c4e14
/avatars/515a49363872c23d57a6f75063606348.svg
Junliang Guo
user
leo-guo
6762c51eeceb9f88f41d1c26
Xinle Cheng
admin_assigned
2024-12-20T10:13:56.564Z
657ab561270ef0b785f21135
/avatars/6a153e8e4d93e32d11d4f537e0159ccf.svg
Xinle Cheng
user
CXL295
6762c51eeceb9f88f41d1c27
Li Song
6762c51eeceb9f88f41d1c28
Jiang Bian
admin_assigned
2024-12-20T10:12:54.696Z
63f253f8f4e30ffd2bd308fb
/avatars/303f4c7ee588f638acf78a7966786e1e.svg
Jiang Bian
user
bianjiang
VidTok: A Versatile and Open-Source Video Tokenizer
Encoding video content into compact latent tokens has become a fundamental
step in video generation and understanding, driven by the need to address the
inherent redundancy in pixel-level representations. Consequently, there is a
growing demand for high-performance, open-source video tokenizers as
video-centric research gains prominence. We introduce VidTok, a versatile video
tokenizer that delivers state-of-the-art performance in both continuous and
discrete tokenizations. VidTok incorporates several key advancements over
existing approaches: 1) model architecture such as convolutional layers and
up/downsampling modules; 2) to address the training instability and codebook
collapse commonly associated with conventional Vector Quantization (VQ), we
integrate Finite Scalar Quantization (FSQ) into discrete video tokenization; 3)
improved training strategies, including a two-stage training process and the
use of reduced frame rates. By integrating these advancements, VidTok achieves
substantial improvements over existing methods, demonstrating superior
performance across multiple metrics, including PSNR, SSIM, LPIPS, and FVD,
under standardized evaluation settings.
VidTok, a state-of-the-art video tokenizer, improves performance in both continuous and discrete tokenization through advanced architecture, Finite Scalar Quantization, and training strategies.
video tokenizer convolutional layers up/downsampling modules Vector Quantization (VQ) Finite Scalar Quantization (FSQ) training stability codebook collapse PSNR SSIM LPIPS FVD