(Almost) Free Modality Stitching of Foundation Models
Foundation multi-modal models are often designed by stitching of multiple
existing pretrained uni-modal models: for example, an image classifier with an
text model. This stitching process is performed by training a connector module
that aims to align the representation spaces of these uni-modal models towards
a multi-modal objective. However, given the complexity of training such
connectors on large scale web-based datasets coupled with the ever-increasing
number of available pretrained uni-modal models, the task of uni-modal models
selection and subsequent connector module training becomes computationally
demanding. To address this under-studied critical problem, we propose
Hypernetwork Model Alignment (Hyma), a novel all-in-one solution for optimal
uni-modal model selection and connector training by leveraging hypernetworks.
Specifically, our framework utilizes the parameter prediction capability of a
hypernetwork to obtain jointly trained connector modules for N times M
combinations of uni-modal models. In our experiments, Hyma reduces the cost of
searching for the best performing uni-modal model pair by 10times, while
matching the ranking and trained connector performance obtained via grid search
across a suite of diverse multi-modal benchmarks.
68798f2021b37e676c8e4050
Jaisidh Singh
68798f2021b37e676c8e4051
Diganta Misra
68798f2021b37e676c8e4052
Boris Knyazev
68798f2021b37e676c8e4053
Antonio Orvieto
(Almost) Free Modality Stitching of Foundation Models
Foundation multi-modal models are often designed by stitching of multiple
existing pretrained uni-modal models: for example, an image classifier with an
text model. This stitching process is performed by training a connector module
that aims to align the representation spaces of these uni-modal models towards
a multi-modal objective. However, given the complexity of training such
connectors on large scale web-based datasets coupled with the ever-increasing
number of available pretrained uni-modal models, the task of uni-modal models
selection and subsequent connector module training becomes computationally
demanding. To address this under-studied critical problem, we propose
Hypernetwork Model Alignment (Hyma), a novel all-in-one solution for optimal
uni-modal model selection and connector training by leveraging hypernetworks.
Specifically, our framework utilizes the parameter prediction capability of a
hypernetwork to obtain jointly trained connector modules for N times M
combinations of uni-modal models. In our experiments, Hyma reduces the cost of
searching for the best performing uni-modal model pair by 10times, while
matching the ranking and trained connector performance obtained via grid search
across a suite of diverse multi-modal benchmarks.
Hypernetwork Model Alignment (Hyma) optimizes uni-modal model selection and connector training for multi-modal models, reducing search costs while maintaining performance.
Hypernetwork Model Alignment Hyma hypernetworks parameter prediction connector modules uni-modal models multi-modal models grid search multi-modal benchmarks