Image Tokenizer Needs Post-Training
Recent image generative models typically capture the image distribution in a
pre-constructed latent space, relying on a frozen image tokenizer. However,
there exists a significant discrepancy between the reconstruction and
generation distribution, where current tokenizers only prioritize the
reconstruction task that happens before generative training without considering
the generation errors during sampling. In this paper, we comprehensively
analyze the reason for this discrepancy in a discrete latent space, and, from
which, we propose a novel tokenizer training scheme including both
main-training and post-training, focusing on improving latent space
construction and decoding respectively. During the main training, a latent
perturbation strategy is proposed to simulate sampling noises, \ie, the
unexpected tokens generated in generative inference. Specifically, we propose a
plug-and-play tokenizer training scheme, which significantly enhances the
robustness of tokenizer, thus boosting the generation quality and convergence
speed, and a novel tokenizer evaluation metric, \ie, pFID, which successfully
correlates the tokenizer performance to generation quality. During
post-training, we further optimize the tokenizer decoder regarding a
well-trained generative model to mitigate the distribution difference between
generated and reconstructed tokens. With a sim400M generator, a discrete
tokenizer trained with our proposed main training achieves a notable 1.60 gFID
and further obtains 1.36 gFID with the additional post-training. Further
experiments are conducted to broadly validate the effectiveness of our
post-training strategy on off-the-shelf discrete and continuous tokenizers,
coupled with autoregressive and diffusion-based generators.
68cc4e043df9ac65e93dc590
Kai Qiu
68cc4e043df9ac65e93dc591
Xiang Li
68cc4e043df9ac65e93dc592
Hao Chen
68cc4e043df9ac65e93dc593
Jason Kuen
68cc4e043df9ac65e93dc594
Xiaohao Xu
68cc4e043df9ac65e93dc595
Jiuxiang Gu
68cc4e043df9ac65e93dc596
Yinyi Luo
68cc4e043df9ac65e93dc597
Bhiksha Raj
68cc4e043df9ac65e93dc598
Zhe Lin
68cc4e043df9ac65e93dc599
Marios Savvides
Image Tokenizer Needs Post-Training
Recent image generative models typically capture the image distribution in a
pre-constructed latent space, relying on a frozen image tokenizer. However,
there exists a significant discrepancy between the reconstruction and
generation distribution, where current tokenizers only prioritize the
reconstruction task that happens before generative training without considering
the generation errors during sampling. In this paper, we comprehensively
analyze the reason for this discrepancy in a discrete latent space, and, from
which, we propose a novel tokenizer training scheme including both
main-training and post-training, focusing on improving latent space
construction and decoding respectively. During the main training, a latent
perturbation strategy is proposed to simulate sampling noises, \ie, the
unexpected tokens generated in generative inference. Specifically, we propose a
plug-and-play tokenizer training scheme, which significantly enhances the
robustness of tokenizer, thus boosting the generation quality and convergence
speed, and a novel tokenizer evaluation metric, \ie, pFID, which successfully
correlates the tokenizer performance to generation quality. During
post-training, we further optimize the tokenizer decoder regarding a
well-trained generative model to mitigate the distribution difference between
generated and reconstructed tokens. With a sim400M generator, a discrete
tokenizer trained with our proposed main training achieves a notable 1.60 gFID
and further obtains 1.36 gFID with the additional post-training. Further
experiments are conducted to broadly validate the effectiveness of our
post-training strategy on off-the-shelf discrete and continuous tokenizers,
coupled with autoregressive and diffusion-based generators.
A novel tokenizer training scheme, including main and post-training phases, improves latent space construction and decoding, enhancing image generation quality and robustness.
latent space image tokenizer reconstruction distribution generation distribution latent perturbation strategy plug-and-play tokenizer training pFID tokenizer evaluation metric tokenizer decoder gFID autoregressive generators diffusion-based generators