Phantom of Latent for Large Language and Vision Models
The success of visual instruction tuning has accelerated the development of
large language and vision models (LLVMs). Following the scaling laws of
instruction-tuned large language models (LLMs), LLVMs either have further
increased their sizes, reaching 26B, 34B, and even 80B parameters. While this
increase in model size has yielded significant performance gains, it demands
substantially more hardware resources for both training and inference.
Consequently, there naturally exists a strong need for efficient LLVMs that
achieve the performance of larger models while being smaller in size. To
achieve this need, we present a new efficient LLVM family with model sizes of
0.5B, 1.8B, 3.8B, and 7B parameters, Phantom, which significantly enhances
learning capabilities within limited structures. By temporarily increasing the
latent hidden dimension during multi-head self-attention (MHSA), we make LLVMs
prepare to look and understand much more vision-language knowledge on the
latent, without substantially increasing physical model sizes. To maximize its
advantage, we introduce Phantom Optimization (PO) using both autoregressive
supervised fine-tuning (SFT) and direct preference optimization (DPO)-like
concept, which effectively follows correct answers while eliminating incorrect
and ambiguous ones. Phantom outperforms numerous larger open- and closed-source
LLVMs, positioning itself as a leading solution in the landscape of efficient
LLVMs.
66f23c6aaef962b23792f12c
Byung-Kwan Lee
claimed_verified
2024-09-24T07:52:40.382Z
657152eb12f162153b50ec9d
https://cdn-avatars.huggingface.co/v1/production/uploads/657152eb12f162153b50ec9d/qnldHP35PclV0pDz_05q8.jpeg
Byung-Kwan Lee
user
BK-Lee
66f23c6aaef962b23792f12d
Sangyun Chung
claimed_verified
2025-01-03T14:06:23.017Z
65a4bf8e90b5e87bcdff41c7
https://cdn-avatars.huggingface.co/v1/production/uploads/65a4bf8e90b5e87bcdff41c7/I6OSoZigV7Fl6OKRl_Yqy.jpeg
Sangyun Chung
user
topyun
66f23c6aaef962b23792f12e
Chae Won Kim
claimed_verified
2024-09-24T07:52:38.696Z
65aa2e84e2a2c86356263340
/avatars/cba9ecceff0f67b2b5dde02e7c0a9991.svg
Chae Won Kim
user
chae-won-kim
66f23c6aaef962b23792f12f
Beomchan Park
66f23c6aaef962b23792f130
Yong Man Ro
Phantom of Latent for Large Language and Vision Models
The success of visual instruction tuning has accelerated the development of
large language and vision models (LLVMs). Following the scaling laws of
instruction-tuned large language models (LLMs), LLVMs either have further
increased their sizes, reaching 26B, 34B, and even 80B parameters. While this
increase in model size has yielded significant performance gains, it demands
substantially more hardware resources for both training and inference.
Consequently, there naturally exists a strong need for efficient LLVMs that
achieve the performance of larger models while being smaller in size. To
achieve this need, we present a new efficient LLVM family with model sizes of
0.5B, 1.8B, 3.8B, and 7B parameters, Phantom, which significantly enhances
learning capabilities within limited structures. By temporarily increasing the
latent hidden dimension during multi-head self-attention (MHSA), we make LLVMs
prepare to look and understand much more vision-language knowledge on the
latent, without substantially increasing physical model sizes. To maximize its
advantage, we introduce Phantom Optimization (PO) using both autoregressive
supervised fine-tuning (SFT) and direct preference optimization (DPO)-like
concept, which effectively follows correct answers while eliminating incorrect
and ambiguous ones. Phantom outperforms numerous larger open- and closed-source
LLVMs, positioning itself as a leading solution in the landscape of efficient
LLVMs.
Phantom, a new efficient LLVM family with smaller model sizes, enhances learning capabilities by temporarily increasing latent hidden dimensions during MHSA and employing Phantom Optimization for improved performance over larger models.
visual instruction tuning large language and vision models (LLVMs) instruction-tuned large language models (LLMs) model sizes training inference efficient LLVMs Phantom multi-head self-attention (MHSA) latent hidden dimension Phantom Optimization (PO) autoregressive supervised fine-tuning (SFT) direct preference optimization (DPO)