Distilling semantically aware orders for autoregressive image generation
Autoregressive patch-based image generation has recently shown competitive
results in terms of image quality and scalability. It can also be easily
integrated and scaled within Vision-Language models. Nevertheless,
autoregressive models require a defined order for patch generation. While a
natural order based on the dictation of the words makes sense for text
generation, there is no inherent generation order that exists for image
generation. Traditionally, a raster-scan order (from top-left to bottom-right)
guides autoregressive image generation models. In this paper, we argue that
this order is suboptimal, as it fails to respect the causality of the image
content: for instance, when conditioned on a visual description of a sunset, an
autoregressive model may generate clouds before the sun, even though the color
of clouds should depend on the color of the sun and not the inverse. In this
work, we show that first by training a model to generate patches in
any-given-order, we can infer both the content and the location (order) of each
patch during generation. Secondly, we use these extracted orders to finetune
the any-given-order model to produce better-quality images. Through our
experiments, we show on two datasets that this new generation method produces
better images than the traditional raster-scan approach, with similar training
costs and no extra annotations.
680b1b33388bb2cfd497ebdb
Rishav Pramanik
claimed_verified
2025-04-25T08:34:27.724Z
62bb84f82ada492aa5775709
https://cdn-avatars.huggingface.co/v1/production/uploads/62bb84f82ada492aa5775709/jv4yKL75t8QzHDHLbhBPT.png
Rishav Pramanik
user
rishavpramanik
680b1b33388bb2cfd497ebdc
Antoine Poupon
claimed_verified
2025-04-26T08:53:34.585Z
65bff50d5604c1d383d23ade
/avatars/92982635885c46e90d4966245d066ef6.svg
Antoine Poupon
user
apoupon
680b1b33388bb2cfd497ebdd
Juan A. Rodriguez
claimed_verified
2025-04-25T12:27:17.091Z
63a614d264f470027818b066
https://cdn-avatars.huggingface.co/v1/production/uploads/63a614d264f470027818b066/Q5Eih2VqD4NaHB_MkNp32.jpeg
Juan A. Rodriguez
user
joanrodai
680b1b33388bb2cfd497ebde
Masih Aminbeidokhti
680b1b33388bb2cfd497ebdf
David Vazquez
claimed_verified
2025-05-08T08:59:53.443Z
62f6691f329d4d014d1b4087
https://cdn-avatars.huggingface.co/v1/production/uploads/1660315866736-noauth.jpeg
David Vazquez
user
dvazquez
680b1b33388bb2cfd497ebe0
Christopher Pal
680b1b33388bb2cfd497ebe1
Zhaozheng Yin
680b1b33388bb2cfd497ebe2
Marco Pedersoli
Distilling semantically aware orders for autoregressive image generation
Autoregressive patch-based image generation has recently shown competitive
results in terms of image quality and scalability. It can also be easily
integrated and scaled within Vision-Language models. Nevertheless,
autoregressive models require a defined order for patch generation. While a
natural order based on the dictation of the words makes sense for text
generation, there is no inherent generation order that exists for image
generation. Traditionally, a raster-scan order (from top-left to bottom-right)
guides autoregressive image generation models. In this paper, we argue that
this order is suboptimal, as it fails to respect the causality of the image
content: for instance, when conditioned on a visual description of a sunset, an
autoregressive model may generate clouds before the sun, even though the color
of clouds should depend on the color of the sun and not the inverse. In this
work, we show that first by training a model to generate patches in
any-given-order, we can infer both the content and the location (order) of each
patch during generation. Secondly, we use these extracted orders to finetune
the any-given-order model to produce better-quality images. Through our
experiments, we show on two datasets that this new generation method produces
better images than the traditional raster-scan approach, with similar training
costs and no extra annotations.
A new method for autoregressive image generation infers patch order during generation, improving image quality over traditional raster-scan approaches without extra annotations.
autoregressive patch-based image generation Vision-Language models raster-scan order causality of image content any-given-order model patch generation