Improving Autoregressive Image Generation through Coarse-to-Fine Token
  Prediction
Autoregressive models have shown remarkable success in image generation by
adapting sequential prediction techniques from language modeling. However,
applying these approaches to images requires discretizing continuous pixel data
through vector quantization methods like VQ-VAE. To alleviate the quantization
errors that existed in VQ-VAE, recent works tend to use larger codebooks.
However, this will accordingly expand vocabulary size, complicating the
autoregressive modeling task. This paper aims to find a way to enjoy the
benefits of large codebooks without making autoregressive modeling more
difficult. Through empirical investigation, we discover that tokens with
similar codeword representations produce similar effects on the final generated
image, revealing significant redundancy in large codebooks. Based on this
insight, we propose to predict tokens from coarse to fine (CTF), realized by
assigning the same coarse label for similar tokens. Our framework consists of
two stages: (1) an autoregressive model that sequentially predicts coarse
labels for each token in the sequence, and (2) an auxiliary model that
simultaneously predicts fine-grained labels for all tokens conditioned on their
coarse labels. Experiments on ImageNet demonstrate our method's superior
performance, achieving an average improvement of 59 points in Inception Score
compared to baselines. Notably, despite adding an inference step, our approach
achieves faster sampling speeds.
67dcf6375fd14aedd3005237
Ziyao Guo
admin_assigned
2025-03-21T13:11:50.727Z
64f1ba1cc640c6d957204786
/avatars/28ede115da5334d2f22a4b5698df1c30.svg
Ziyao Guo
user
ZiyaoGuo
67dcf6375fd14aedd3005238
Kaipeng Zhang
admin_assigned
2025-03-21T13:12:11.686Z
65f1713552c38a91e0a445e8
/avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
kaipeng
user
kpzhang996
67dcf6375fd14aedd3005239
Michael Qizhe Shieh
admin_assigned
2025-03-21T13:12:23.239Z
64d48e9ef2a3941669810f23
/avatars/495e6785abc44d1facfcf385edd6a222.svg
Michael Shieh
user
Michael2008S
Improving Autoregressive Image Generation through Coarse-to-Fine Token
  Prediction
Autoregressive models have shown remarkable success in image generation by
adapting sequential prediction techniques from language modeling. However,
applying these approaches to images requires discretizing continuous pixel data
through vector quantization methods like VQ-VAE. To alleviate the quantization
errors that existed in VQ-VAE, recent works tend to use larger codebooks.
However, this will accordingly expand vocabulary size, complicating the
autoregressive modeling task. This paper aims to find a way to enjoy the
benefits of large codebooks without making autoregressive modeling more
difficult. Through empirical investigation, we discover that tokens with
similar codeword representations produce similar effects on the final generated
image, revealing significant redundancy in large codebooks. Based on this
insight, we propose to predict tokens from coarse to fine (CTF), realized by
assigning the same coarse label for similar tokens. Our framework consists of
two stages: (1) an autoregressive model that sequentially predicts coarse
labels for each token in the sequence, and (2) an auxiliary model that
simultaneously predicts fine-grained labels for all tokens conditioned on their
coarse labels. Experiments on ImageNet demonstrate our method's superior
performance, achieving an average improvement of 59 points in Inception Score
compared to baselines. Notably, despite adding an inference step, our approach
achieves faster sampling speeds.
A coarse-to-fine approach improves image generation by optimizing autoregressive modeling with large codebooks, enhancing performance and speed.
autoregressive models image generation VQ-VAE vector quantization coarse-to-fine Inception Score