Unified Model for Image, Video, Audio and Language Tasks
Large Language Models (LLMs) have made the ambitious quest for generalist
agents significantly far from being a fantasy. A key hurdle for building such
general models is the diversity and heterogeneity of tasks and modalities. A
promising solution is unification, allowing the support of a myriad of tasks
and modalities within one unified framework. While few large models (e.g.,
Flamingo (Alayrac et al., 2022), trained on massive datasets, can support more
than two modalities, current small to mid-scale unified models are still
limited to 2 modalities, usually image-text or video-text. The question that we
ask is: is it possible to build efficiently a unified model that can support
all modalities? To answer this, we propose UnIVAL, a step further towards this
ambitious goal. Without relying on fancy datasets sizes or models with billions
of parameters, the ~ 0.25B parameter UnIVAL model goes beyond two modalities
and unifies text, images, video, and audio into a single model. Our model is
efficiently pretrained on many tasks, based on task balancing and multimodal
curriculum learning. UnIVAL shows competitive performance to existing
state-of-the-art approaches, across image and video-text tasks. The feature
representations learned from image and video-text modalities, allows the model
to achieve competitive performance when finetuned on audio-text tasks, despite
not being pretrained on audio. Thanks to the unified model, we propose a novel
study on multimodal model merging via weight interpolation of models trained on
different multimodal tasks, showing their benefits in particular for
out-of-distribution generalization. Finally, we motivate unification by showing
the synergy between tasks. The model weights and code are released here:
https://github.com/mshukor/UnIVAL.
64c867ecd11ed5b1e2df7441
Mustafa Shukor
admin_assigned
2023-08-01T10:00:42.255Z
62bdeedd01dc22b4d22a371e
/avatars/3cc0643feb53bf2e895ec12c275d5483.svg
Mustafa Shukor
user
mshukor
64c867ecd11ed5b1e2df7442
Corentin Dancette
admin_assigned
2023-08-01T10:00:57.915Z
62cdea59a9be5c195561c2b8
/avatars/959b702e57718b9029634cb41772dcef.svg
Corentin Dancette
user
cdancette
64c867ecd11ed5b1e2df7443
Alexandre Rame
admin_assigned
2023-08-01T10:01:18.163Z
63c94ede00104ea998de19a6
/avatars/273959d87f0c67747588cf0700d64039.svg
Alexandre Rame
user
alexrame
64c867ecd11ed5b1e2df7444
Matthieu Cord
Unified Model for Image, Video, Audio and Language Tasks
Large Language Models (LLMs) have made the ambitious quest for generalist
agents significantly far from being a fantasy. A key hurdle for building such
general models is the diversity and heterogeneity of tasks and modalities. A
promising solution is unification, allowing the support of a myriad of tasks
and modalities within one unified framework. While few large models (e.g.,
Flamingo (Alayrac et al., 2022), trained on massive datasets, can support more
than two modalities, current small to mid-scale unified models are still
limited to 2 modalities, usually image-text or video-text. The question that we
ask is: is it possible to build efficiently a unified model that can support
all modalities? To answer this, we propose UnIVAL, a step further towards this
ambitious goal. Without relying on fancy datasets sizes or models with billions
of parameters, the ~ 0.25B parameter UnIVAL model goes beyond two modalities
and unifies text, images, video, and audio into a single model. Our model is
efficiently pretrained on many tasks, based on task balancing and multimodal
curriculum learning. UnIVAL shows competitive performance to existing
state-of-the-art approaches, across image and video-text tasks. The feature
representations learned from image and video-text modalities, allows the model
to achieve competitive performance when finetuned on audio-text tasks, despite
not being pretrained on audio. Thanks to the unified model, we propose a novel
study on multimodal model merging via weight interpolation of models trained on
different multimodal tasks, showing their benefits in particular for
out-of-distribution generalization. Finally, we motivate unification by showing
the synergy between tasks. The model weights and code are released here:
https://github.com/mshukor/UnIVAL.
UnIVAL, a unified model with 0.25B parameters, efficiently supports text, images, video, and audio, demonstrating competitive performance and benefiting out-of-distribution generalization through multimodal weight interpolation.
Large Language Models generalist agents unification modalities Flamingo UnIVAL pretrained efficient multimodal curriculum learning feature representations weight interpolation out-of-distribution generalization