Autoregressive Models in Vision: A Survey
Autoregressive modeling has been a huge success in the field of natural
language processing (NLP). Recently, autoregressive models have emerged as a
significant area of focus in computer vision, where they excel in producing
high-quality visual content. Autoregressive models in NLP typically operate on
subword tokens. However, the representation strategy in computer vision can
vary in different levels, i.e., pixel-level, token-level, or
scale-level, reflecting the diverse and hierarchical nature of visual data
compared to the sequential structure of language. This survey comprehensively
examines the literature on autoregressive models applied to vision. To improve
readability for researchers from diverse research backgrounds, we start with
preliminary sequence representation and modeling in vision. Next, we divide the
fundamental frameworks of visual autoregressive models into three general
sub-categories, including pixel-based, token-based, and scale-based models
based on the strategy of representation. We then explore the interconnections
between autoregressive models and other generative models. Furthermore, we
present a multi-faceted categorization of autoregressive models in computer
vision, including image generation, video generation, 3D generation, and
multi-modal generation. We also elaborate on their applications in diverse
domains, including emerging domains such as embodied AI and 3D medical AI, with
about 250 related references. Finally, we highlight the current challenges to
autoregressive models in vision with suggestions about potential research
directions. We have also set up a Github repository to organize the papers
included in this survey at:
https://github.com/ChaofanTao/Autoregressive-Models-in-Vision-Survey.
673395a92f0f0d8e0ec5af19
Jing Xiong
673395a92f0f0d8e0ec5af1a
Gongye Liu
claimed_verified
2025-02-05T13:37:45.786Z
6553316bf151de82f6a23e1d
https://cdn-avatars.huggingface.co/v1/production/uploads/6553316bf151de82f6a23e1d/GTBkSj4Fa3OoyM6Muz_Sc.jpeg
Gongye Liu
user
liuhuohuo
673395a92f0f0d8e0ec5af1b
Lun Huang
673395a92f0f0d8e0ec5af1c
Chengyue Wu
673395a92f0f0d8e0ec5af1d
Taiqiang Wu
claimed_verified
2025-05-26T08:45:08.397Z
6621cea88850e38ffbb1854f
https://cdn-avatars.huggingface.co/v1/production/uploads/6621cea88850e38ffbb1854f/LeytEEjSwnnqB-zFN1Tgt.jpeg
Taki WU
user
taki555
673395a92f0f0d8e0ec5af1e
Yao Mu
673395a92f0f0d8e0ec5af1f
Yuan Yao
673395a92f0f0d8e0ec5af20
Hui Shen
claimed_verified
2025-09-15T15:10:35.159Z
640feb91b0ee289c8583dd59
https://cdn-avatars.huggingface.co/v1/production/uploads/640feb91b0ee289c8583dd59/CA4UZj2mEsJO2UR0NnN5H.jpeg
Hui Shen
user
Cloudriver
673395a92f0f0d8e0ec5af21
Zhongwei Wan
673395a92f0f0d8e0ec5af22
Jinfa Huang
claimed_verified
2025-07-09T08:54:43.056Z
63f37af60be81bdc5d92eebb
/avatars/b8dfdff4ab36988ec9a8643e82a3d2db.svg
Huang
user
Jinfa
673395a92f0f0d8e0ec5af23
Chaofan Tao
673395a92f0f0d8e0ec5af24
Shen Yan
673395a92f0f0d8e0ec5af25
Huaxiu Yao
673395a92f0f0d8e0ec5af26
Lingpeng Kong
673395a92f0f0d8e0ec5af27
Hongxia Yang
673395a92f0f0d8e0ec5af28
Mi Zhang
673395a92f0f0d8e0ec5af29
Guillermo Sapiro
673395a92f0f0d8e0ec5af2a
Jiebo Luo
673395a92f0f0d8e0ec5af2b
Ping Luo
673395a92f0f0d8e0ec5af2c
Ngai Wong
Autoregressive Models in Vision: A Survey
Autoregressive modeling has been a huge success in the field of natural
language processing (NLP). Recently, autoregressive models have emerged as a
significant area of focus in computer vision, where they excel in producing
high-quality visual content. Autoregressive models in NLP typically operate on
subword tokens. However, the representation strategy in computer vision can
vary in different levels, i.e., pixel-level, token-level, or
scale-level, reflecting the diverse and hierarchical nature of visual data
compared to the sequential structure of language. This survey comprehensively
examines the literature on autoregressive models applied to vision. To improve
readability for researchers from diverse research backgrounds, we start with
preliminary sequence representation and modeling in vision. Next, we divide the
fundamental frameworks of visual autoregressive models into three general
sub-categories, including pixel-based, token-based, and scale-based models
based on the strategy of representation. We then explore the interconnections
between autoregressive models and other generative models. Furthermore, we
present a multi-faceted categorization of autoregressive models in computer
vision, including image generation, video generation, 3D generation, and
multi-modal generation. We also elaborate on their applications in diverse
domains, including emerging domains such as embodied AI and 3D medical AI, with
about 250 related references. Finally, we highlight the current challenges to
autoregressive models in vision with suggestions about potential research
directions. We have also set up a Github repository to organize the papers
included in this survey at:
https://github.com/ChaofanTao/Autoregressive-Models-in-Vision-Survey.
This survey explores the application of autoregressive models in computer vision, categorizing them into pixel-based, token-based, and scale-based models, and discussing their integration with other generative models and applications in emerging domains.
autoregressive modeling pixel-level token-level scale-level visual autoregressive models image generation video generation 3D generation multi-modal generation embodied AI 3D medical AI