Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV
  Cache Compression
Visual Autoregressive (VAR) modeling has garnered significant attention for
its innovative next-scale prediction approach, which yields substantial
improvements in efficiency, scalability, and zero-shot generalization.
Nevertheless, the coarse-to-fine methodology inherent in VAR results in
exponential growth of the KV cache during inference, causing considerable
memory consumption and computational redundancy. To address these bottlenecks,
we introduce ScaleKV, a novel KV cache compression framework tailored for VAR
architectures. ScaleKV leverages two critical observations: varying cache
demands across transformer layers and distinct attention patterns at different
scales. Based on these insights, ScaleKV categorizes transformer layers into
two functional groups: drafters and refiners. Drafters exhibit dispersed
attention across multiple scales, thereby requiring greater cache capacity.
Conversely, refiners focus attention on the current token map to process local
details, consequently necessitating substantially reduced cache capacity.
ScaleKV optimizes the multi-scale inference pipeline by identifying
scale-specific drafters and refiners, facilitating differentiated cache
management tailored to each scale. Evaluation on the state-of-the-art
text-to-image VAR model family, Infinity, demonstrates that our approach
effectively reduces the required KV cache memory to 10% while preserving
pixel-level fidelity.
6835239e7309025530c85ba3
Kunjun Li
claimed_verified
2025-05-27T07:51:58.885Z
6540ef0e733c1ce6a6fc989a
https://cdn-avatars.huggingface.co/v1/production/uploads/6540ef0e733c1ce6a6fc989a/lyDLbmJ-h4nUmkWZCvWtg.jpeg
Kunjun Li
user
stargazerx0
6835239e7309025530c85ba4
Zigeng Chen
claimed_verified
2025-05-27T15:50:12.945Z
65811eeaa2284a018e51f1ba
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/dH8UZj6Kk5HJkI1DItCNm.jpeg
Zigeng Chen
user
Zigeng
6835239e7309025530c85ba5
Cheng-Yen Yang
6835239e7309025530c85ba6
Jenq-Neng Hwang
Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV
  Cache Compression
Visual Autoregressive (VAR) modeling has garnered significant attention for
its innovative next-scale prediction approach, which yields substantial
improvements in efficiency, scalability, and zero-shot generalization.
Nevertheless, the coarse-to-fine methodology inherent in VAR results in
exponential growth of the KV cache during inference, causing considerable
memory consumption and computational redundancy. To address these bottlenecks,
we introduce ScaleKV, a novel KV cache compression framework tailored for VAR
architectures. ScaleKV leverages two critical observations: varying cache
demands across transformer layers and distinct attention patterns at different
scales. Based on these insights, ScaleKV categorizes transformer layers into
two functional groups: drafters and refiners. Drafters exhibit dispersed
attention across multiple scales, thereby requiring greater cache capacity.
Conversely, refiners focus attention on the current token map to process local
details, consequently necessitating substantially reduced cache capacity.
ScaleKV optimizes the multi-scale inference pipeline by identifying
scale-specific drafters and refiners, facilitating differentiated cache
management tailored to each scale. Evaluation on the state-of-the-art
text-to-image VAR model family, Infinity, demonstrates that our approach
effectively reduces the required KV cache memory to 10% while preserving
pixel-level fidelity.
ScaleKV compresses the KV cache in Visual Autoregressive models by differentiating drafters and refiners across transformer layers, reducing memory consumption while maintaining high fidelity.
Visual Autoregressive VAR KV cache transformer layers drafters refiners memory consumption Infinity pixel-level fidelity