Meteor: Mamba-based Traversal of Rationale for Large Language and Vision
  Models
The rapid development of large language and vision models (LLVMs) has been
driven by advances in visual instruction tuning. Recently, open-source LLVMs
have curated high-quality visual instruction tuning datasets and utilized
additional vision encoders or multiple computer vision models in order to
narrow the performance gap with powerful closed-source LLVMs. These
advancements are attributed to multifaceted information required for diverse
capabilities, including fundamental image understanding, real-world knowledge
about common-sense and non-object concepts (e.g., charts, diagrams, symbols,
signs, and math problems), and step-by-step procedures for solving complex
questions. Drawing from the multifaceted information, we present a new
efficient LLVM, Mamba-based traversal of rationales (Meteor), which leverages
multifaceted rationale to enhance understanding and answering capabilities. To
embed lengthy rationales containing abundant information, we employ the Mamba
architecture, capable of processing sequential data with linear time
complexity. We introduce a new concept of traversal of rationale that
facilitates efficient embedding of rationale. Subsequently, the backbone
multimodal language model (MLM) is trained to generate answers with the aid of
rationale. Through these steps, Meteor achieves significant improvements in
vision language performances across multiple evaluation benchmarks requiring
diverse capabilities, without scaling up the model size or employing additional
vision encoders and computer vision models.
6653f0c6607894ea80484b14
Byung-Kwan Lee
claimed_verified
2024-05-27T07:17:58.143Z
657152eb12f162153b50ec9d
https://cdn-avatars.huggingface.co/v1/production/uploads/657152eb12f162153b50ec9d/qnldHP35PclV0pDz_05q8.jpeg
Byung-Kwan Lee
user
BK-Lee
6653f0c6607894ea80484b15
Chae Won Kim
admin_assigned
2024-05-27T07:59:47.261Z
65aa2e84e2a2c86356263340
/avatars/cba9ecceff0f67b2b5dde02e7c0a9991.svg
Chae Won Kim
user
chae-won-kim
6653f0c6607894ea80484b16
Beomchan Park
claimed_verified
2024-05-27T07:17:54.134Z
65390e7ce279d2fc80a3f800
/avatars/a930c1f20fa1eef372ca92139cb32887.svg
Beomchan Park
user
bpbpbp0810
6653f0c6607894ea80484b17
Yong Man Ro
admin_assigned
2024-05-27T07:59:55.451Z
660603529e3555d648e3baf1
/avatars/0f54479afcfc19df00b25d5aedb4cf67.svg
Yong Man Ro
user
dwightro
Meteor: Mamba-based Traversal of Rationale for Large Language and Vision
  Models
The rapid development of large language and vision models (LLVMs) has been
driven by advances in visual instruction tuning. Recently, open-source LLVMs
have curated high-quality visual instruction tuning datasets and utilized
additional vision encoders or multiple computer vision models in order to
narrow the performance gap with powerful closed-source LLVMs. These
advancements are attributed to multifaceted information required for diverse
capabilities, including fundamental image understanding, real-world knowledge
about common-sense and non-object concepts (e.g., charts, diagrams, symbols,
signs, and math problems), and step-by-step procedures for solving complex
questions. Drawing from the multifaceted information, we present a new
efficient LLVM, Mamba-based traversal of rationales (Meteor), which leverages
multifaceted rationale to enhance understanding and answering capabilities. To
embed lengthy rationales containing abundant information, we employ the Mamba
architecture, capable of processing sequential data with linear time
complexity. We introduce a new concept of traversal of rationale that
facilitates efficient embedding of rationale. Subsequently, the backbone
multimodal language model (MLM) is trained to generate answers with the aid of
rationale. Through these steps, Meteor achieves significant improvements in
vision language performances across multiple evaluation benchmarks requiring
diverse capabilities, without scaling up the model size or employing additional
vision encoders and computer vision models.
Meteor, an efficient large language and vision model, enhances understanding and answering capabilities by embedding multifaceted rationales using the Mamba architecture, leading to improved vision language performance without increasing model size or using additional vision encoders.
large language and vision models visual instruction tuning datasets vision encoders computer vision models image understanding real-world knowledge Mamba architecture traversal of rationales multimodal language model vision language performance