Beyond Next-Token: Next-X Prediction for Autoregressive Visual
  Generation
Autoregressive (AR) modeling, known for its next-token prediction paradigm,
underpins state-of-the-art language and visual generative models.
Traditionally, a ``token'' is treated as the smallest prediction unit, often a
discrete symbol in language or a quantized patch in vision. However, the
optimal token definition for 2D image structures remains an open question.
Moreover, AR models suffer from exposure bias, where teacher forcing during
training leads to error accumulation at inference. In this paper, we propose
xAR, a generalized AR framework that extends the notion of a token to an entity
X, which can represent an individual patch token, a cell (a ktimes k
grouping of neighboring patches), a subsample (a non-local grouping of distant
patches), a scale (coarse-to-fine resolution), or even a whole image.
Additionally, we reformulate discrete token classification as
continuous entity regression, leveraging flow-matching methods at each
AR step. This approach conditions training on noisy entities instead of ground
truth tokens, leading to Noisy Context Learning, which effectively alleviates
exposure bias. As a result, xAR offers two key advantages: (1) it enables
flexible prediction units that capture different contextual granularity and
spatial structures, and (2) it mitigates exposure bias by avoiding reliance on
teacher forcing. On ImageNet-256 generation benchmark, our base model, xAR-B
(172M), outperforms DiT-XL/SiT-XL (675M) while achieving 20times faster
inference. Meanwhile, xAR-H sets a new state-of-the-art with an FID of 1.24,
running 2.2times faster than the previous best-performing model without
relying on vision foundation modules (\eg, DINOv2) or advanced guidance
interval sampling.
67c1643aa4ccbde471532ba6
Sucheng Ren
67c1643aa4ccbde471532ba7
Qihang Yu
67c1643aa4ccbde471532ba8
Ju He
claimed_verified
2025-03-17T08:45:43.724Z
661c9059bcd78151e5c06ea1
https://cdn-avatars.huggingface.co/v1/production/uploads/661c9059bcd78151e5c06ea1/27bfNo1LZeZQ77vWuAa10.png
Ju He
user
turkeyju
67c1643aa4ccbde471532ba9
Xiaohui Shen
67c1643aa4ccbde471532baa
Alan Yuille
67c1643aa4ccbde471532bab
Liang-Chieh Chen
Beyond Next-Token: Next-X Prediction for Autoregressive Visual
  Generation
Autoregressive (AR) modeling, known for its next-token prediction paradigm,
underpins state-of-the-art language and visual generative models.
Traditionally, a ``token'' is treated as the smallest prediction unit, often a
discrete symbol in language or a quantized patch in vision. However, the
optimal token definition for 2D image structures remains an open question.
Moreover, AR models suffer from exposure bias, where teacher forcing during
training leads to error accumulation at inference. In this paper, we propose
xAR, a generalized AR framework that extends the notion of a token to an entity
X, which can represent an individual patch token, a cell (a ktimes k
grouping of neighboring patches), a subsample (a non-local grouping of distant
patches), a scale (coarse-to-fine resolution), or even a whole image.
Additionally, we reformulate discrete token classification as
continuous entity regression, leveraging flow-matching methods at each
AR step. This approach conditions training on noisy entities instead of ground
truth tokens, leading to Noisy Context Learning, which effectively alleviates
exposure bias. As a result, xAR offers two key advantages: (1) it enables
flexible prediction units that capture different contextual granularity and
spatial structures, and (2) it mitigates exposure bias by avoiding reliance on
teacher forcing. On ImageNet-256 generation benchmark, our base model, xAR-B
(172M), outperforms DiT-XL/SiT-XL (675M) while achieving 20times faster
inference. Meanwhile, xAR-H sets a new state-of-the-art with an FID of 1.24,
running 2.2times faster than the previous best-performing model without
relying on vision foundation modules (\eg, DINOv2) or advanced guidance
interval sampling.
xAR enhances generative models by extending token definitions and using continuous entity regression with flow-matching to improve prediction flexibility and reduce exposure bias.
autoregressive models next-token prediction token patch cell subsample scale continuous entity regression flow-matching noisy context learning exposure bias ImageNet-256 FID inference speed DiT-XL SiT-XL DINOv2 guidance interval sampling