Matryoshka Multimodal Models
Large Multimodal Models (LMMs) such as LLaVA have shown strong performance in
visual-linguistic reasoning. These models first embed images into a fixed large
number of visual tokens and then feed them into a Large Language Model (LLM).
However, this design causes an excessive number of tokens for dense visual
scenarios such as high-resolution images and videos, leading to great
inefficiency. While token pruning/merging methods do exist, they produce a
single length output for each image and do not afford flexibility in trading
off information density v.s. efficiency. Inspired by the concept of Matryoshka
Dolls, we propose M3: Matryoshka Multimodal Models, which learns to represent
visual content as nested sets of visual tokens that capture information across
multiple coarse-to-fine granularities. Our approach offers several unique
benefits for LMMs: (1) One can explicitly control the visual granularity per
test instance during inference, e.g. , adjusting the number of tokens used to
represent an image based on the anticipated complexity or simplicity of the
content; (2) M3 provides a framework for analyzing the granularity needed for
existing datasets, where we find that COCO-style benchmarks only need around ~9
visual tokens to obtain accuracy similar to that of using all 576 tokens; (3)
Our approach provides a foundation to explore the best trade-off between
performance and visual token length at sample level, where our investigation
reveals that a large gap exists between the oracle upper bound and current
fixed-scale representations.
665547a7e6ddaa4f89ad2020
Mu Cai
admin_assigned
2024-05-28T07:46:24.511Z
63b7b2c6bd2d153522821766
https://cdn-avatars.huggingface.co/v1/production/uploads/63b7b2c6bd2d153522821766/aHtga-_OUdOrg_TRrXO08.jpeg
Mu Cai
user
mucai
665547a7e6ddaa4f89ad2021
Jianwei Yang
admin_assigned
2024-05-28T08:12:03.879Z
6125df7f25027fb1ea9c7a41
https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
Jianwei Yang
user
jw2yang
665547a7e6ddaa4f89ad2022
Jianfeng Gao
admin_assigned
2024-05-28T08:12:12.457Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
665547a7e6ddaa4f89ad2023
Yong Jae Lee
admin_assigned
2024-05-28T08:12:18.996Z
649f41ee70a478f8b36b2984
/avatars/1c9a76717a450ac4aeb62a1e823d2e4a.svg
Yong Jae Lee
user
yjlee0222
Matryoshka Multimodal Models
Large Multimodal Models (LMMs) such as LLaVA have shown strong performance in
visual-linguistic reasoning. These models first embed images into a fixed large
number of visual tokens and then feed them into a Large Language Model (LLM).
However, this design causes an excessive number of tokens for dense visual
scenarios such as high-resolution images and videos, leading to great
inefficiency. While token pruning/merging methods do exist, they produce a
single length output for each image and do not afford flexibility in trading
off information density v.s. efficiency. Inspired by the concept of Matryoshka
Dolls, we propose M3: Matryoshka Multimodal Models, which learns to represent
visual content as nested sets of visual tokens that capture information across
multiple coarse-to-fine granularities. Our approach offers several unique
benefits for LMMs: (1) One can explicitly control the visual granularity per
test instance during inference, e.g. , adjusting the number of tokens used to
represent an image based on the anticipated complexity or simplicity of the
content; (2) M3 provides a framework for analyzing the granularity needed for
existing datasets, where we find that COCO-style benchmarks only need around ~9
visual tokens to obtain accuracy similar to that of using all 576 tokens; (3)
Our approach provides a foundation to explore the best trade-off between
performance and visual token length at sample level, where our investigation
reveals that a large gap exists between the oracle upper bound and current
fixed-scale representations.
Multimodal Models (LMMs) LLaVA visual-linguistic reasoning visual tokens Large Language Model (LLM) token pruning token merging Matryoshka Dolls Matryoshka Multimodal Models (M3) nested sets of visual tokens coarse-to-fine granularities visual granularity inference COPYO-style benchmarks