UniTok: A Unified Tokenizer for Visual Generation and Understanding
The representation disparity between visual generation and understanding
imposes a critical gap in integrating these capabilities into a single
framework. To bridge this gap, we introduce UniTok, a discrete visual tokenizer
that encodes fine-grained details for generation while also capturing
high-level semantics for understanding. Despite recent studies have shown that
these objectives could induce loss conflicts in training, we reveal that the
underlying bottleneck stems from limited representational capacity of discrete
tokens. We address this by introducing multi-codebook quantization, which
divides vector quantization with several independent sub-codebooks to expand
the latent feature space, while avoiding training instability caused by
overlarge codebooks. Our method significantly raises the upper limit of unified
discrete tokenizers to match or even surpass domain-specific continuous
tokenizers. For instance, UniTok achieves a remarkable rFID of 0.38 (versus
0.87 for SD-VAE) and a zero-shot accuracy of 78.6% (versus 76.2% for CLIP) on
ImageNet. Our code is available at https://github.com/FoundationVision/UniTok.
67c13c68d8247a49b808fdac
Chuofan Ma
admin_assigned
2025-02-28T12:31:29.232Z
62c585eb09baf76938a70de8
/avatars/ae8cca53710b3325bf0dd0f08c2b1bbf.svg
Chuofan Ma
user
cfma
67c13c68d8247a49b808fdad
Yi Jiang
claimed_verified
2025-03-02T20:18:07.566Z
6344dcb1cd37e44d9ed46508
https://cdn-avatars.huggingface.co/v1/production/uploads/6344dcb1cd37e44d9ed46508/J92UKSxKR3iziD2WJfih4.jpeg
Yi Jiang
user
JiangYi
67c13c68d8247a49b808fdae
Junfeng Wu
admin_assigned
2025-02-28T12:31:35.467Z
6572ac949a5c2d6df9fab3c5
/avatars/b5d70a86a452198381eee1c8f513ceec.svg
Junfeng Wu
user
JunfengWu
67c13c68d8247a49b808fdaf
Jihan Yang
admin_assigned
2025-02-28T12:31:41.522Z
6304baf041387c7f1177a5d2
/avatars/795c63f2394080eec78ca7981d4a1f78.svg
Jihan Yang
user
jihanyang
67c13c68d8247a49b808fdb0
Xin Yu
claimed_verified
2025-03-12T15:26:27.595Z
62d1a71ac40bde8b986c3748
/avatars/6dbfba55563c71ed0db0d59c2629d525.svg
Xin Yu
user
Andyx
67c13c68d8247a49b808fdb1
Zehuan Yuan
admin_assigned
2025-02-28T12:32:06.178Z
661a80af3557013b638061d5
/avatars/4c551aeb223e257a5fc45b5b6c7ded49.svg
Zehuan Yuan
user
sweetrabor
67c13c68d8247a49b808fdb2
Bingyue Peng
67c13c68d8247a49b808fdb3
Xiaojuan Qi
claimed_verified
2025-07-15T19:14:01.394Z
6875266f9cd3191dfddc7071
/avatars/64c581910833b111e9a7bae5b8740229.svg
xiaojuan qi
user
xjqi
UniTok: A Unified Tokenizer for Visual Generation and Understanding
The representation disparity between visual generation and understanding
imposes a critical gap in integrating these capabilities into a single
framework. To bridge this gap, we introduce UniTok, a discrete visual tokenizer
that encodes fine-grained details for generation while also capturing
high-level semantics for understanding. Despite recent studies have shown that
these objectives could induce loss conflicts in training, we reveal that the
underlying bottleneck stems from limited representational capacity of discrete
tokens. We address this by introducing multi-codebook quantization, which
divides vector quantization with several independent sub-codebooks to expand
the latent feature space, while avoiding training instability caused by
overlarge codebooks. Our method significantly raises the upper limit of unified
discrete tokenizers to match or even surpass domain-specific continuous
tokenizers. For instance, UniTok achieves a remarkable rFID of 0.38 (versus
0.87 for SD-VAE) and a zero-shot accuracy of 78.6% (versus 76.2% for CLIP) on
ImageNet. Our code is available at https://github.com/FoundationVision/UniTok.
UniTok, a discrete visual tokenizer utilizing multi-codebook quantization, achieves competitive performance in image generation and understanding tasks.
discrete visual tokenizer fine-grained details high-level semantics multi-codebook quantization vector quantization latent feature space representational capacity unified discrete tokenizers domain-specific continuous tokenizers rFID zero-shot accuracy ImageNet