Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale
  Prediction
We present Visual AutoRegressive modeling (VAR), a new generation paradigm
that redefines the autoregressive learning on images as coarse-to-fine
"next-scale prediction" or "next-resolution prediction", diverging from the
standard raster-scan "next-token prediction". This simple, intuitive
methodology allows autoregressive (AR) transformers to learn visual
distributions fast and generalize well: VAR, for the first time, makes AR
models surpass diffusion transformers in image generation. On ImageNet 256x256
benchmark, VAR significantly improve AR baseline by improving Frechet inception
distance (FID) from 18.65 to 1.80, inception score (IS) from 80.4 to 356.4,
with around 20x faster inference speed. It is also empirically verified that
VAR outperforms the Diffusion Transformer (DiT) in multiple dimensions
including image quality, inference speed, data efficiency, and scalability.
Scaling up VAR models exhibits clear power-law scaling laws similar to those
observed in LLMs, with linear correlation coefficients near -0.998 as solid
evidence. VAR further showcases zero-shot generalization ability in downstream
tasks including image in-painting, out-painting, and editing. These results
suggest VAR has initially emulated the two important properties of LLMs:
Scaling Laws and zero-shot task generalization. We have released all models and
codes to promote the exploration of AR/VAR models for visual generation and
unified learning.
660e129262d63ad000c149de
Keyu Tian
claimed_verified
2024-04-04T07:55:43.864Z
60e73ffd06ad9ae5bbcfc52c
https://cdn-avatars.huggingface.co/v1/production/uploads/1625767908838-noauth.jpeg
Keyu Tian
user
keyu-tian
660e129262d63ad000c149df
Yi Jiang
claimed_verified
2024-04-05T10:05:39.069Z
6344dcb1cd37e44d9ed46508
https://cdn-avatars.huggingface.co/v1/production/uploads/6344dcb1cd37e44d9ed46508/J92UKSxKR3iziD2WJfih4.jpeg
Yi Jiang
user
JiangYi
660e129262d63ad000c149e0
Zehuan Yuan
admin_assigned
2024-04-25T07:38:11.619Z
661a80af3557013b638061d5
/avatars/4c551aeb223e257a5fc45b5b6c7ded49.svg
Zehuan Yuan
user
sweetrabor
660e129262d63ad000c149e1
Bingyue Peng
660e129262d63ad000c149e2
Liwei Wang
admin_assigned
2024-04-25T07:38:33.015Z
6497272b6401266eb6752b89
/avatars/d94a011660f4cad5344756027840dc30.svg
Liwei Wang
user
asdfg80
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale
  Prediction
We present Visual AutoRegressive modeling (VAR), a new generation paradigm
that redefines the autoregressive learning on images as coarse-to-fine
"next-scale prediction" or "next-resolution prediction", diverging from the
standard raster-scan "next-token prediction". This simple, intuitive
methodology allows autoregressive (AR) transformers to learn visual
distributions fast and generalize well: VAR, for the first time, makes AR
models surpass diffusion transformers in image generation. On ImageNet 256x256
benchmark, VAR significantly improve AR baseline by improving Frechet inception
distance (FID) from 18.65 to 1.80, inception score (IS) from 80.4 to 356.4,
with around 20x faster inference speed. It is also empirically verified that
VAR outperforms the Diffusion Transformer (DiT) in multiple dimensions
including image quality, inference speed, data efficiency, and scalability.
Scaling up VAR models exhibits clear power-law scaling laws similar to those
observed in LLMs, with linear correlation coefficients near -0.998 as solid
evidence. VAR further showcases zero-shot generalization ability in downstream
tasks including image in-painting, out-painting, and editing. These results
suggest VAR has initially emulated the two important properties of LLMs:
Scaling Laws and zero-shot task generalization. We have released all models and
codes to promote the exploration of AR/VAR models for visual generation and
unified learning.
Visual AutoRegressive (VAR) modeling improves image generation performance and speed over autoregressive and diffusion transformer models, achieving superior results in image quality, data efficiency, and scalability.
Visual AutoRegressive modeling autoregressive learning next-scale prediction next-resolution prediction autoregressive transformers diffusion transformers Frechet inception distance inception score image quality inference speed data efficiency scalability LLMs scaling laws zero-shot generalization image in-painting out-painting image editing