LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact
  Language Model
We train a suite of multimodal foundation models (MMFM) using the popular
LLaVA framework with the recently released Gemma family of large language
models (LLMs). Of particular interest is the 2B parameter Gemma model, which
provides opportunities to construct capable small-scale MMFMs. In line with
findings from other papers in this space, we test the effect of ablating three
design features: pretraining the connector, utilizing a more powerful image
backbone, and increasing the size of the language backbone. The resulting
models, which we call LLaVA-Gemma, exhibit moderate performance on an array of
evaluations, but fail to improve past the current comparably sized SOTA models.
Closer analysis of performance shows mixed effects; skipping pretraining tends
to reduce performance, larger vision models sometimes improve performance, and
increasing language model size has inconsistent effects. We publicly release
training recipes, code and weights for our models for the LLaVA-Gemma models.
660cbcd9852e9d33f1810e1a
Musashi Hinck
claimed_verified
2024-04-04T07:50:02.643Z
65de48ac04b9352580c257fb
/avatars/9fd28805926c06927cea1140e7c0440e.svg
Musashi Hinck
user
musashihinck
660cbcd9852e9d33f1810e1b
Matthew L. Olson
660cbcd9852e9d33f1810e1c
David Cobbley
claimed_verified
2024-04-04T07:50:14.811Z
6515c52426224cf8b1e9f5ca
/avatars/38eeac1e8a4dd37152c556af60814c65.svg
Dave Cobbley
user
djcobble
660cbcd9852e9d33f1810e1d
Shao-Yen Tseng
660cbcd9852e9d33f1810e1e
Vasudev Lal
LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact
  Language Model
We train a suite of multimodal foundation models (MMFM) using the popular
LLaVA framework with the recently released Gemma family of large language
models (LLMs). Of particular interest is the 2B parameter Gemma model, which
provides opportunities to construct capable small-scale MMFMs. In line with
findings from other papers in this space, we test the effect of ablating three
design features: pretraining the connector, utilizing a more powerful image
backbone, and increasing the size of the language backbone. The resulting
models, which we call LLaVA-Gemma, exhibit moderate performance on an array of
evaluations, but fail to improve past the current comparably sized SOTA models.
Closer analysis of performance shows mixed effects; skipping pretraining tends
to reduce performance, larger vision models sometimes improve performance, and
increasing language model size has inconsistent effects. We publicly release
training recipes, code and weights for our models for the LLaVA-Gemma models.
LLaVA-Gemma models, trained with varying design features, achieve moderate performance relative to state-of-the-art models with similar sizes.
multimodal foundation models LLaVA Gemma family parameter-efficient fine-tuning image backbone language backbone