Scaling Laws for Native Multimodal Models Scaling Laws for Native
  Multimodal Models
Building general-purpose models that can effectively perceive the world
through multimodal signals has been a long-standing goal. Current approaches
involve integrating separately pre-trained components, such as connecting
vision encoders to LLMs and continuing multimodal training. While such
approaches exhibit remarkable sample efficiency, it remains an open question
whether such late-fusion architectures are inherently superior. In this work,
we revisit the architectural design of native multimodal models (NMMs)--those
trained from the ground up on all modalities--and conduct an extensive scaling
laws study, spanning 457 trained models with different architectures and
training mixtures. Our investigation reveals no inherent advantage to
late-fusion architectures over early-fusion ones, which do not rely on image
encoders. On the contrary, early-fusion exhibits stronger performance at lower
parameter counts, is more efficient to train, and is easier to deploy.
Motivated by the strong performance of the early-fusion architectures, we show
that incorporating Mixture of Experts (MoEs) allows for models that learn
modality-specific weights, significantly enhancing performance.
67f8e4acc7b30121f9c054d1
Mustafa Shukor
claimed_verified
2025-04-22T10:14:19.280Z
62bdeedd01dc22b4d22a371e
/avatars/3cc0643feb53bf2e895ec12c275d5483.svg
Mustafa Shukor
user
mshukor
67f8e4acc7b30121f9c054d2
Enrico Fini
67f8e4acc7b30121f9c054d3
Victor Guilherme Turrisi da Costa
67f8e4acc7b30121f9c054d4
Matthieu Cord
67f8e4acc7b30121f9c054d5
Joshua Susskind
67f8e4acc7b30121f9c054d6
Alaaeldin El-Nouby
Scaling Laws for Native Multimodal Models Scaling Laws for Native
  Multimodal Models
Building general-purpose models that can effectively perceive the world
through multimodal signals has been a long-standing goal. Current approaches
involve integrating separately pre-trained components, such as connecting
vision encoders to LLMs and continuing multimodal training. While such
approaches exhibit remarkable sample efficiency, it remains an open question
whether such late-fusion architectures are inherently superior. In this work,
we revisit the architectural design of native multimodal models (NMMs)--those
trained from the ground up on all modalities--and conduct an extensive scaling
laws study, spanning 457 trained models with different architectures and
training mixtures. Our investigation reveals no inherent advantage to
late-fusion architectures over early-fusion ones, which do not rely on image
encoders. On the contrary, early-fusion exhibits stronger performance at lower
parameter counts, is more efficient to train, and is easier to deploy.
Motivated by the strong performance of the early-fusion architectures, we show
that incorporating Mixture of Experts (MoEs) allows for models that learn
modality-specific weights, significantly enhancing performance.
Early-fusion architectures, which do not rely on image encoders, outperform late-fusion models in native multimodal architectures, especially when enhanced with Mixture of Experts (MoEs).
native multimodal models early-fusion late-fusion Mixture of Experts