A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive
  Transformer for Efficient Finegrained Image Generation
This work tackles the information loss bottleneck of vector-quantization (VQ)
autoregressive image generation by introducing a novel model architecture
called the 2-Dimensional Autoregression (DnD) Transformer. The DnD-Transformer
predicts more codes for an image by introducing a new autoregression direction,
model depth, along with the sequence length direction. Compared to
traditional 1D autoregression and previous work utilizing similar 2D image
decomposition such as RQ-Transformer, the DnD-Transformer is an end-to-end
model that can generate higher quality images with the same backbone model size
and sequence length, opening a new optimization perspective for autoregressive
image generation. Furthermore, our experiments reveal that the
DnD-Transformer's potential extends beyond generating natural images. It can
even generate images with rich text and graphical elements in a self-supervised
manner, demonstrating an understanding of these combined modalities. This has
not been previously demonstrated for popular vision generative models such as
diffusion models, showing a spark of vision-language intelligence when trained
solely on images. Code, datasets and models are open at
https://github.com/chenllliang/DnD-Transformer.
67052598e2d3403eaf1fc3bc
Liang Chen
extracted_pending
2024-10-08T12:29:20.810Z
61b0a4ce1b3d95b3d1ed9251
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Wwjr26vdudX5KYVTb8Q0a.png
Liang Chen
user
leonardPKU
67052598e2d3403eaf1fc3bd
Sinan Tan
67052598e2d3403eaf1fc3be
Zefan Cai
admin_assigned
2024-10-09T11:21:21.768Z
64b15284372d4340772a3dca
/avatars/417d5f1bc1bcb5e4d5de6169673c2cf7.svg
Zefan Cai
user
ZefanCai
67052598e2d3403eaf1fc3bf
Weichu Xie
67052598e2d3403eaf1fc3c0
Haozhe Zhao
claimed_verified
2025-05-26T08:45:50.853Z
63ff09a02098b9ad105a09f6
/avatars/4409ca5d320050cf4c3df05962c7ff58.svg
Hans Zhao
user
BleachNick
67052598e2d3403eaf1fc3c1
Yichi Zhang
67052598e2d3403eaf1fc3c2
Junyang Lin
admin_assigned
2024-10-09T11:43:10.132Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
67052598e2d3403eaf1fc3c3
Jinze Bai
admin_assigned
2024-10-09T11:43:20.221Z
60113fad51e116b62cd0a30e
/avatars/469357d0a4a5d2e104ae5e32801b395d.svg
Jinze Bai
user
Jinze
67052598e2d3403eaf1fc3c4
Tianyu Liu
67052598e2d3403eaf1fc3c5
Baobao Chang
A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive
  Transformer for Efficient Finegrained Image Generation
This work tackles the information loss bottleneck of vector-quantization (VQ)
autoregressive image generation by introducing a novel model architecture
called the 2-Dimensional Autoregression (DnD) Transformer. The DnD-Transformer
predicts more codes for an image by introducing a new autoregression direction,
model depth, along with the sequence length direction. Compared to
traditional 1D autoregression and previous work utilizing similar 2D image
decomposition such as RQ-Transformer, the DnD-Transformer is an end-to-end
model that can generate higher quality images with the same backbone model size
and sequence length, opening a new optimization perspective for autoregressive
image generation. Furthermore, our experiments reveal that the
DnD-Transformer's potential extends beyond generating natural images. It can
even generate images with rich text and graphical elements in a self-supervised
manner, demonstrating an understanding of these combined modalities. This has
not been previously demonstrated for popular vision generative models such as
diffusion models, showing a spark of vision-language intelligence when trained
solely on images. Code, datasets and models are open at
https://github.com/chenllliang/DnD-Transformer.
The DnD-Transformer improves autoregressive image generation by introducing model depth autoregression, leading to higher quality images and the ability to generate images with textual and graphical elements.
2-Dimensional Autoregression DnD Transformer vector-quantization autoregressive autoregression direction RQ Transformer end-to-end model vision generative models diffusion models vision-language intelligence