Efficient LLaMA-3.2-Vision by Trimming Cross-attended Visual Features
Visual token reduction lowers inference costs caused by extensive image
features in large vision-language models (LVLMs). Unlike relevant studies that
prune tokens in self-attention-only LVLMs, our work uniquely addresses
cross-attention-based models, which achieve superior performance. We identify
that the key-value (KV) cache size for image tokens in cross-attention layers
significantly exceeds that of text tokens in self-attention layers, posing a
major compute bottleneck. To mitigate this issue, we exploit the sparse nature
in cross-attention maps to selectively prune redundant visual features. Our
Trimmed Llama effectively reduces KV cache demands without requiring additional
training. By benefiting from 50%-reduced visual features, our model can reduce
inference latency and memory usage while achieving benchmark parity.
67eca9a501a4d1c29e4e70f1
Jewon Lee
admin_assigned
2025-04-02T12:22:32.687Z
642d395784bf892b8fa91816
/avatars/2db2202641ad7ed9aeaea7e8bfc3acca.svg
Jewon Lee
user
je1lee
67eca9a501a4d1c29e4e70f2
Ki-Ung Song
admin_assigned
2025-04-02T12:22:38.520Z
645f6407b5a2706bce578eed
/avatars/8f576ebd8b7c89578e083ccabf6ebc33.svg
Ki-Ung song
user
sk851
67eca9a501a4d1c29e4e70f3
Seungmin Yang
admin_assigned
2025-04-02T12:22:46.031Z
679cedca09f47bf5c1ca251f
/avatars/aa8d6cc5f2993849b7f7c16d2cb1c8f6.svg
Seungmin Yang
user
ainoobchobo
67eca9a501a4d1c29e4e70f4
Donguk Lim
admin_assigned
2025-04-02T12:22:55.363Z
62e7773c870b23dac1b944e2
/avatars/919ad4150b0b0aa58bc49651b4911c53.svg
Lim
user
DongUK
67eca9a501a4d1c29e4e70f5
Jaeyeon Kim
admin_assigned
2025-04-02T12:23:09.888Z
6548c84e10d47c1f771f183a
/avatars/37d759063d870e2fe0df926efebff750.svg
Jaeyeon Kim
user
jaeyeonkim99
67eca9a501a4d1c29e4e70f6
Wooksu Shin
admin_assigned
2025-04-02T12:23:19.454Z
67a0a0d48f047b67c3172e9e
/avatars/fa2ee986071f3642c8c7cd6ff693dbfd.svg
Wooksu Shin
user
dnrtn1101
67eca9a501a4d1c29e4e70f7
Bo-Kyeong Kim
admin_assigned
2025-04-02T12:23:24.647Z
64374f10cd93f4c9a34c1239
https://cdn-avatars.huggingface.co/v1/production/uploads/64374f10cd93f4c9a34c1239/tdkdcgl18rKXLvyTEhfnG.jpeg
Bo-Kyeong Kim
user
bokyeong1015
67eca9a501a4d1c29e4e70f8
Yong Jae Lee
admin_assigned
2025-04-02T12:23:30.906Z
649f41ee70a478f8b36b2984
/avatars/1c9a76717a450ac4aeb62a1e823d2e4a.svg
Yong Jae Lee
user
yjlee0222
67eca9a501a4d1c29e4e70f9
Tae-Ho Kim
admin_assigned
2025-04-02T12:23:44.160Z
67a701f26b44b5a72af20c9e
https://cdn-avatars.huggingface.co/v1/production/uploads/67a701f26b44b5a72af20c9e/vriqmqgRG3f1mMIjj3tce.png
Tae-ho Kim
user
sabioguru
Efficient LLaMA-3.2-Vision by Trimming Cross-attended Visual Features
Visual token reduction lowers inference costs caused by extensive image
features in large vision-language models (LVLMs). Unlike relevant studies that
prune tokens in self-attention-only LVLMs, our work uniquely addresses
cross-attention-based models, which achieve superior performance. We identify
that the key-value (KV) cache size for image tokens in cross-attention layers
significantly exceeds that of text tokens in self-attention layers, posing a
major compute bottleneck. To mitigate this issue, we exploit the sparse nature
in cross-attention maps to selectively prune redundant visual features. Our
Trimmed Llama effectively reduces KV cache demands without requiring additional
training. By benefiting from 50%-reduced visual features, our model can reduce
inference latency and memory usage while achieving benchmark parity.
Visual token reduction in cross-attention-based models improves inference efficiency by pruning redundant features without additional training, maintaining benchmark performance.
key-value cache cross-attention visual features inference latency memory usage benchmark parity Trimmed Llama