Stabilize the Latent Space for Image Autoregressive Modeling: A Unified
  Perspective
Latent-based image generative models, such as Latent Diffusion Models (LDMs)
and Mask Image Models (MIMs), have achieved notable success in image generation
tasks. These models typically leverage reconstructive autoencoders like VQGAN
or VAE to encode pixels into a more compact latent space and learn the data
distribution in the latent space instead of directly from pixels. However, this
practice raises a pertinent question: Is it truly the optimal choice? In
response, we begin with an intriguing observation: despite sharing the same
latent space, autoregressive models significantly lag behind LDMs and MIMs in
image generation. This finding contrasts sharply with the field of NLP, where
the autoregressive model GPT has established a commanding presence. To address
this discrepancy, we introduce a unified perspective on the relationship
between latent space and generative models, emphasizing the stability of latent
space in image generative modeling. Furthermore, we propose a simple but
effective discrete image tokenizer to stabilize the latent space for image
generative modeling. Experimental results show that image autoregressive
modeling with our tokenizer (DiGIT) benefits both image understanding and image
generation with the next token prediction principle, which is inherently
straightforward for GPT models but challenging for other generative models.
Remarkably, for the first time, a GPT-style autoregressive model for images
outperforms LDMs, which also exhibits substantial improvement akin to GPT when
scaling up model size. Our findings underscore the potential of an optimized
latent space and the integration of discrete tokenization in advancing the
capabilities of image generative models. The code is available at
https://github.com/DAMO-NLP-SG/DiGIT.
6710bfb58bbf77ae7996d0e9
Yongxin Zhu
claimed_verified
2024-11-16T19:47:15.032Z
637c7d8a88699fba70e1e1ff
/avatars/e7f60fab6ca9ad5ecd320bd7cd51ce2e.svg
yongxinzhu
user
youngsheen
6710bfb58bbf77ae7996d0ea
Bocheng Li
6710bfb58bbf77ae7996d0eb
Hang Zhang
6710bfb58bbf77ae7996d0ec
Xin Li
claimed_verified
2024-10-18T08:29:39.670Z
63913b120cf6b11c487ca31d
/avatars/aec44edd5470dd6e767e0a25efd6fb5d.svg
Xin Li
user
lixin4ever
6710bfb58bbf77ae7996d0ed
Linli Xu
6710bfb58bbf77ae7996d0ee
Lidong Bing
claimed_verified
2024-10-25T09:31:11.681Z
6454685a548f22be598414c4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
Lidong Bing
user
LidongBing
Stabilize the Latent Space for Image Autoregressive Modeling: A Unified
  Perspective
Latent-based image generative models, such as Latent Diffusion Models (LDMs)
and Mask Image Models (MIMs), have achieved notable success in image generation
tasks. These models typically leverage reconstructive autoencoders like VQGAN
or VAE to encode pixels into a more compact latent space and learn the data
distribution in the latent space instead of directly from pixels. However, this
practice raises a pertinent question: Is it truly the optimal choice? In
response, we begin with an intriguing observation: despite sharing the same
latent space, autoregressive models significantly lag behind LDMs and MIMs in
image generation. This finding contrasts sharply with the field of NLP, where
the autoregressive model GPT has established a commanding presence. To address
this discrepancy, we introduce a unified perspective on the relationship
between latent space and generative models, emphasizing the stability of latent
space in image generative modeling. Furthermore, we propose a simple but
effective discrete image tokenizer to stabilize the latent space for image
generative modeling. Experimental results show that image autoregressive
modeling with our tokenizer (DiGIT) benefits both image understanding and image
generation with the next token prediction principle, which is inherently
straightforward for GPT models but challenging for other generative models.
Remarkably, for the first time, a GPT-style autoregressive model for images
outperforms LDMs, which also exhibits substantial improvement akin to GPT when
scaling up model size. Our findings underscore the potential of an optimized
latent space and the integration of discrete tokenization in advancing the
capabilities of image generative models. The code is available at
https://github.com/DAMO-NLP-SG/DiGIT.
A discrete image tokenizer stabilizes latent space for image generative modeling, enabling GPT-style autoregressive models to outperform Latent Diffusion Models.
Latent Diffusion Models Mask Image Models VQGAN VAE latent space image generative modeling autoregressive models image understanding image generation DiGIT