When Less is Enough: Adaptive Token Reduction for Efficient Image
  Representation
Vision encoders typically generate a large number of visual tokens, providing
information-rich representations but significantly increasing computational
demands. This raises the question of whether all generated tokens are equally
valuable or if some of them can be discarded to reduce computational costs
without compromising quality. In this paper, we introduce a new method for
determining feature utility based on the idea that less valuable features can
be reconstructed from more valuable ones. We implement this concept by
integrating an autoencoder with a Gumbel-Softmax selection mechanism, that
allows identifying and retaining only the most informative visual tokens. To
validate our approach, we compared the performance of the LLaVA-NeXT model,
using features selected by our method with randomly selected features. We found
that on OCR-based tasks, more than 50% of the visual context can be removed
with minimal performance loss, whereas randomly discarding the same proportion
of features significantly affects the model capabilities. Furthermore, in
general-domain tasks, even randomly retaining only 30% of tokens achieves
performance comparable to using the full set of visual tokens. Our results
highlight a promising direction towards adaptive and efficient multimodal
pruning that facilitates scalable and low-overhead inference without
compromising performance.
67e0ffb029682c8065e1c223
Eduard Allakhverdov
claimed_verified
2025-06-10T09:30:34.777Z
6437d0a951c7ebfc813c735b
/avatars/6cbac4e4be5029655702c5d8b9046b90.svg
Allakhverdov Eduard
user
combat-helicopter
67e0ffb029682c8065e1c224
Elizaveta Goncharova
admin_assigned
2025-03-24T12:38:44.495Z
6310ff34bc152fa3e810c186
/avatars/bfd63bcd81548283f5e496e3693bf143.svg
Elizaveta Goncharova
user
Elizaveta
67e0ffb029682c8065e1c225
Andrey Kuznetsov
claimed_verified
2025-03-24T14:49:31.067Z
643984dceb7c5616ef3f5d54
https://cdn-avatars.huggingface.co/v1/production/uploads/643984dceb7c5616ef3f5d54/10JRkblrRIEVci6UJwvPz.jpeg
Andrey Kuznetsov
user
kuznetsoffandrey
When Less is Enough: Adaptive Token Reduction for Efficient Image
  Representation
Vision encoders typically generate a large number of visual tokens, providing
information-rich representations but significantly increasing computational
demands. This raises the question of whether all generated tokens are equally
valuable or if some of them can be discarded to reduce computational costs
without compromising quality. In this paper, we introduce a new method for
determining feature utility based on the idea that less valuable features can
be reconstructed from more valuable ones. We implement this concept by
integrating an autoencoder with a Gumbel-Softmax selection mechanism, that
allows identifying and retaining only the most informative visual tokens. To
validate our approach, we compared the performance of the LLaVA-NeXT model,
using features selected by our method with randomly selected features. We found
that on OCR-based tasks, more than 50% of the visual context can be removed
with minimal performance loss, whereas randomly discarding the same proportion
of features significantly affects the model capabilities. Furthermore, in
general-domain tasks, even randomly retaining only 30% of tokens achieves
performance comparable to using the full set of visual tokens. Our results
highlight a promising direction towards adaptive and efficient multimodal
pruning that facilitates scalable and low-overhead inference without
compromising performance.
A method using autoencoder and Gumbel-Softmax selection identifies and retains only the most informative visual tokens, enabling efficient multimodal pruning with minimal performance loss.
autoencoder Gumbel-Softmax selection mechanism visual tokens feature utility informative visual tokens LLaVA-NeXT OCR-based tasks general-domain tasks multimodal pruning scalable inference low-overhead inference