Jointly Training Large Autoregressive Multimodal Models
In recent years, advances in the large-scale pretraining of language and
text-to-image models have revolutionized the field of machine learning. Yet,
integrating these two modalities into a single, robust model capable of
generating seamless multimodal outputs remains a significant challenge. To
address this gap, we present the Joint Autoregressive Mixture (JAM) framework,
a modular approach that systematically fuses existing text and image generation
models. We also introduce a specialized, data-efficient instruction-tuning
strategy, tailored for mixed-modal generation tasks. Our final instruct-tuned
model demonstrates unparalleled performance in generating high-quality
multimodal outputs and represents the first model explicitly designed for this
purpose.
651501516a451f1caf345611
Emanuele Aiello
claimed_verified
2024-06-15T21:28:49.848Z
6515ac7f60757b8c8f6b8d31
/avatars/ed033f30f9f78ed68da976e6d01af0ea.svg
Emanuele Aiello
user
Ema97x
651501516a451f1caf345612
Lili Yu
admin_assigned
2023-09-28T09:19:30.861Z
6466d867f3e78d1d6be05da0
/avatars/4a2458f9fe5e1637798adfaf01cfa4fa.svg
Lili Yu
user
lilisierrayu
651501516a451f1caf345613
Yixin Nie
admin_assigned
2023-09-28T09:19:42.319Z
5f7919d5f1e7ef6e919a1f55
/avatars/fb747fb43b65ccdfb08d7bf71525f92f.svg
Yixin Nie
user
ynie
651501516a451f1caf345614
Armen Aghajanyan
2023-09-28T09:19:53.408Z
651501516a451f1caf345615
Barlas Oguz
admin_assigned
2023-09-28T09:19:59.971Z
6317cb5b212fce5a3cd97227
/avatars/c2b4eefa7c17cdf2299c07466c361ab0.svg
Barlas Oguz
user
barlaso
Jointly Training Large Autoregressive Multimodal Models
In recent years, advances in the large-scale pretraining of language and
text-to-image models have revolutionized the field of machine learning. Yet,
integrating these two modalities into a single, robust model capable of
generating seamless multimodal outputs remains a significant challenge. To
address this gap, we present the Joint Autoregressive Mixture (JAM) framework,
a modular approach that systematically fuses existing text and image generation
models. We also introduce a specialized, data-efficient instruction-tuning
strategy, tailored for mixed-modal generation tasks. Our final instruct-tuned
model demonstrates unparalleled performance in generating high-quality
multimodal outputs and represents the first model explicitly designed for this
purpose.
The Joint Autoregressive Mixture (JAM) framework integrates text and image generation models for high-quality multimodal output, exceeding previous methods with a specialized instruction-tuning strategy.
Joint Autoregressive Mixture JAM framework multimodal outputs instruction-tuning text generation image generation