Accelerating Auto-regressive Text-to-Image Generation with Training-free
  Speculative Jacobi Decoding
The current large auto-regressive models can generate high-quality,
high-resolution images, but these models require hundreds or even thousands of
steps of next-token prediction during inference, resulting in substantial time
consumption. In existing studies, Jacobi decoding, an iterative parallel
decoding algorithm, has been used to accelerate the auto-regressive generation
and can be executed without training. However, the Jacobi decoding relies on a
deterministic criterion to determine the convergence of iterations. Thus, it
works for greedy decoding but is incompatible with sampling-based decoding
which is crucial for visual quality and diversity in the current
auto-regressive text-to-image generation. In this paper, we propose a
training-free probabilistic parallel decoding algorithm, Speculative Jacobi
Decoding (SJD), to accelerate auto-regressive text-to-image generation. By
introducing a probabilistic convergence criterion, our SJD accelerates the
inference of auto-regressive text-to-image generation while maintaining the
randomness in sampling-based token decoding and allowing the model to generate
diverse images. Specifically, SJD facilitates the model to predict multiple
tokens at each step and accepts tokens based on the probabilistic criterion,
enabling the model to generate images with fewer steps than the conventional
next-token-prediction paradigm. We also investigate the token initialization
strategies that leverage the spatial locality of visual data to further improve
the acceleration ratio under specific scenarios. We conduct experiments for our
proposed SJD on multiple auto-regressive text-to-image generation models,
showing the effectiveness of model acceleration without sacrificing the visual
quality.
67020787400ad7197b4b4a5f
Yao Teng
claimed_verified
2024-10-07T07:26:55.869Z
6427e08288215cee63b1c44d
https://cdn-avatars.huggingface.co/v1/production/uploads/6427e08288215cee63b1c44d/rzaG978FF-ywzicWNl_xl.jpeg
yao teng
user
tytyt
67020787400ad7197b4b4a60
Han Shi
claimed_verified
2024-10-13T20:18:10.248Z
637751ad7169ed981703d26a
/avatars/9823f115040a3d6b0240038010e46cb6.svg
Han Shi
user
shihan96
67020787400ad7197b4b4a61
Xian Liu
claimed_verified
2024-10-30T09:55:47.337Z
6423f10a3fa01ecba6fca861
/avatars/e4b4c7dddc5cf962ade45d8bdfa10b1a.svg
Xian Liu
user
alvinliu0
67020787400ad7197b4b4a62
Xuefei Ning
claimed_verified
2024-11-15T09:43:47.048Z
641031b1a78453b8d96b8420
https://cdn-avatars.huggingface.co/v1/production/uploads/1678782881444-noauth.jpeg
Xuefei Ning
user
Foxfi
67020787400ad7197b4b4a63
Guohao Dai
67020787400ad7197b4b4a64
Yu Wang
67020787400ad7197b4b4a65
Zhenguo Li
67020787400ad7197b4b4a66
Xihui Liu
claimed_verified
2024-10-16T09:44:29.460Z
65d5ec74cd05bc1eaa125040
/avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
Xihui Liu
user
XihuiLiu
Accelerating Auto-regressive Text-to-Image Generation with Training-free
  Speculative Jacobi Decoding
The current large auto-regressive models can generate high-quality,
high-resolution images, but these models require hundreds or even thousands of
steps of next-token prediction during inference, resulting in substantial time
consumption. In existing studies, Jacobi decoding, an iterative parallel
decoding algorithm, has been used to accelerate the auto-regressive generation
and can be executed without training. However, the Jacobi decoding relies on a
deterministic criterion to determine the convergence of iterations. Thus, it
works for greedy decoding but is incompatible with sampling-based decoding
which is crucial for visual quality and diversity in the current
auto-regressive text-to-image generation. In this paper, we propose a
training-free probabilistic parallel decoding algorithm, Speculative Jacobi
Decoding (SJD), to accelerate auto-regressive text-to-image generation. By
introducing a probabilistic convergence criterion, our SJD accelerates the
inference of auto-regressive text-to-image generation while maintaining the
randomness in sampling-based token decoding and allowing the model to generate
diverse images. Specifically, SJD facilitates the model to predict multiple
tokens at each step and accepts tokens based on the probabilistic criterion,
enabling the model to generate images with fewer steps than the conventional
next-token-prediction paradigm. We also investigate the token initialization
strategies that leverage the spatial locality of visual data to further improve
the acceleration ratio under specific scenarios. We conduct experiments for our
proposed SJD on multiple auto-regressive text-to-image generation models,
showing the effectiveness of model acceleration without sacrificing the visual
quality.
A new probabilistic parallel decoding algorithm, Speculative Jacobi Decoding, accelerates auto-regressive text-to-image generation while maintaining visual quality and diversity.
auto-regressive models high-resolution images next-token prediction Jacobi decoding probabilistic parallel decoding Speculative Jacobi Decoding sampling-based decoding token initialization spatial locality auto-regressive text-to-image generation