Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient
In the rapidly advancing field of image generation, Visual Auto-Regressive
(VAR) modeling has garnered considerable attention for its innovative
next-scale prediction approach. This paradigm offers substantial improvements
in efficiency, scalability, and zero-shot generalization. Yet, the inherently
coarse-to-fine nature of VAR introduces a prolonged token sequence, leading to
prohibitive memory consumption and computational redundancies. To address these
bottlenecks, we propose Collaborative Decoding (CoDe), a novel efficient
decoding strategy tailored for the VAR framework. CoDe capitalizes on two
critical observations: the substantially reduced parameter demands at larger
scales and the exclusive generation patterns across different scales. Based on
these insights, we partition the multi-scale inference process into a seamless
collaboration between a large model and a small model. The large model serves
as the 'drafter', specializing in generating low-frequency content at smaller
scales, while the smaller model serves as the 'refiner', solely focusing on
predicting high-frequency details at larger scales. This collaboration yields
remarkable efficiency with minimal impact on quality: CoDe achieves a 1.7x
speedup, slashes memory usage by around 50%, and preserves image quality with
only a negligible FID increase from 1.95 to 1.98. When drafting steps are
further decreased, CoDe can achieve an impressive 2.9x acceleration ratio,
reaching 41 images/s at 256x256 resolution on a single NVIDIA 4090 GPU, while
preserving a commendable FID of 2.27. The code is available at
https://github.com/czg1225/CoDe
6747d429cd2486b8b0e1fb24
Zigeng Chen
claimed_verified
2024-11-28T09:19:44.162Z
65811eeaa2284a018e51f1ba
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/dH8UZj6Kk5HJkI1DItCNm.jpeg
Zigeng Chen
user
Zigeng
6747d429cd2486b8b0e1fb25
Xinyin Ma
claimed_verified
2024-11-28T09:19:42.545Z
64396ebc21221ac7411852b3
https://cdn-avatars.huggingface.co/v1/production/uploads/64396ebc21221ac7411852b3/SR0dC8N0bdj9tZFxYPpSf.jpeg
Xinyin Ma
user
horseee
6747d429cd2486b8b0e1fb26
Gongfan Fang
claimed_verified
2025-04-16T13:42:16.302Z
646a1939c37ca1e12308fe81
/avatars/752e9d86018e7d33ad8bcd741203fd86.svg
Gongfan Fang
user
Vinnnf
6747d429cd2486b8b0e1fb27
Xinchao Wang
Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient
In the rapidly advancing field of image generation, Visual Auto-Regressive
(VAR) modeling has garnered considerable attention for its innovative
next-scale prediction approach. This paradigm offers substantial improvements
in efficiency, scalability, and zero-shot generalization. Yet, the inherently
coarse-to-fine nature of VAR introduces a prolonged token sequence, leading to
prohibitive memory consumption and computational redundancies. To address these
bottlenecks, we propose Collaborative Decoding (CoDe), a novel efficient
decoding strategy tailored for the VAR framework. CoDe capitalizes on two
critical observations: the substantially reduced parameter demands at larger
scales and the exclusive generation patterns across different scales. Based on
these insights, we partition the multi-scale inference process into a seamless
collaboration between a large model and a small model. The large model serves
as the 'drafter', specializing in generating low-frequency content at smaller
scales, while the smaller model serves as the 'refiner', solely focusing on
predicting high-frequency details at larger scales. This collaboration yields
remarkable efficiency with minimal impact on quality: CoDe achieves a 1.7x
speedup, slashes memory usage by around 50%, and preserves image quality with
only a negligible FID increase from 1.95 to 1.98. When drafting steps are
further decreased, CoDe can achieve an impressive 2.9x acceleration ratio,
reaching 41 images/s at 256x256 resolution on a single NVIDIA 4090 GPU, while
preserving a commendable FID of 2.27. The code is available at
https://github.com/czg1225/CoDe
Collaborative Decoding (CoDe) enhances Visual Auto-Regressive (VAR) modeling efficiency by integrating a large and small model to reduce memory usage and computation time without significantly affecting image quality.
Visual Auto-Regressive (VAR) modeling Collaborative Decoding (CoDe) multi-scale inference large model small model drafter refiner low-frequency content high-frequency details speedup memory usage FID NVIDIA 4090 GPU