Token-Shuffle: Towards High-Resolution Image Generation with
  Autoregressive Models
Autoregressive (AR) models, long dominant in language generation, are
increasingly applied to image synthesis but are often considered less
competitive than Diffusion-based models. A primary limitation is the
substantial number of image tokens required for AR models, which constrains
both training and inference efficiency, as well as image resolution. To address
this, we present Token-Shuffle, a novel yet simple method that reduces the
number of image tokens in Transformer. Our key insight is the dimensional
redundancy of visual vocabularies in Multimodal Large Language Models (MLLMs),
where low-dimensional visual codes from visual encoder are directly mapped to
high-dimensional language vocabularies. Leveraging this, we consider two key
operations: token-shuffle, which merges spatially local tokens along channel
dimension to decrease the input token number, and token-unshuffle, which
untangles the inferred tokens after Transformer blocks to restore the spatial
arrangement for output. Jointly training with textual prompts, our strategy
requires no additional pretrained text-encoder and enables MLLMs to support
extremely high-resolution image synthesis in a unified next-token prediction
way while maintaining efficient training and inference. For the first time, we
push the boundary of AR text-to-image generation to a resolution of 2048x2048
with gratifying generation performance. In GenAI-benchmark, our 2.7B model
achieves 0.77 overall score on hard prompts, outperforming AR models LlamaGen
by 0.18 and diffusion models LDM by 0.15. Exhaustive large-scale human
evaluations also demonstrate our prominent image generation ability in terms of
text-alignment, visual flaw, and visual appearance. We hope that Token-Shuffle
can serve as a foundational design for efficient high-resolution image
generation within MLLMs.
680b318bbbebf87944bc9595
Xu Ma
680b318bbbebf87944bc9596
Peize Sun
680b318bbbebf87944bc9597
Haoyu Ma
680b318bbbebf87944bc9598
Hao Tang
680b318bbbebf87944bc9599
Chih-Yao Ma
680b318bbbebf87944bc959a
Jialiang Wang
680b318bbbebf87944bc959b
Kunpeng Li
680b318bbbebf87944bc959c
Xiaoliang Dai
680b318bbbebf87944bc959d
Yujun Shi
680b318bbbebf87944bc959e
Xuan Ju
680b318bbbebf87944bc959f
Yushi Hu
680b318bbbebf87944bc95a0
Artsiom Sanakoyeu
680b318bbbebf87944bc95a1
Felix Juefei-Xu
680b318bbbebf87944bc95a2
Ji Hou
680b318bbbebf87944bc95a3
Junjiao Tian
680b318bbbebf87944bc95a4
Tao Xu
680b318bbbebf87944bc95a5
Tingbo Hou
680b318bbbebf87944bc95a6
Yen-Cheng Liu
680b318bbbebf87944bc95a7
Zecheng He
680b318bbbebf87944bc95a8
Zijian He
680b318bbbebf87944bc95a9
Matt Feiszli
680b318bbbebf87944bc95aa
Peizhao Zhang
680b318bbbebf87944bc95ab
Peter Vajda
680b318bbbebf87944bc95ac
Sam Tsai
680b318bbbebf87944bc95ad
Yun Fu
Token-Shuffle: Towards High-Resolution Image Generation with
  Autoregressive Models
Autoregressive (AR) models, long dominant in language generation, are
increasingly applied to image synthesis but are often considered less
competitive than Diffusion-based models. A primary limitation is the
substantial number of image tokens required for AR models, which constrains
both training and inference efficiency, as well as image resolution. To address
this, we present Token-Shuffle, a novel yet simple method that reduces the
number of image tokens in Transformer. Our key insight is the dimensional
redundancy of visual vocabularies in Multimodal Large Language Models (MLLMs),
where low-dimensional visual codes from visual encoder are directly mapped to
high-dimensional language vocabularies. Leveraging this, we consider two key
operations: token-shuffle, which merges spatially local tokens along channel
dimension to decrease the input token number, and token-unshuffle, which
untangles the inferred tokens after Transformer blocks to restore the spatial
arrangement for output. Jointly training with textual prompts, our strategy
requires no additional pretrained text-encoder and enables MLLMs to support
extremely high-resolution image synthesis in a unified next-token prediction
way while maintaining efficient training and inference. For the first time, we
push the boundary of AR text-to-image generation to a resolution of 2048x2048
with gratifying generation performance. In GenAI-benchmark, our 2.7B model
achieves 0.77 overall score on hard prompts, outperforming AR models LlamaGen
by 0.18 and diffusion models LDM by 0.15. Exhaustive large-scale human
evaluations also demonstrate our prominent image generation ability in terms of
text-alignment, visual flaw, and visual appearance. We hope that Token-Shuffle
can serve as a foundational design for efficient high-resolution image
generation within MLLMs.
A Token-Shuffle method enhances high-resolution AR image generation by reducing token numbers in Transformers, enabling efficient generation and outperforming diffusion models.
Autoregressive models Token-Shuffle token-shuffle token-unshuffle Multimodal Large Language Models visual vocabularies visual codes Transformer next-token prediction hard prompts GenAI-benchmark LlamaGen LDM text-alignment visual flaw visual appearance