Zipper: A Multi-Tower Decoder Architecture for Fusing Modalities
Integrating multiple generative foundation models, especially those trained
on different modalities, into something greater than the sum of its parts poses
significant challenges. Two key hurdles are the availability of aligned data
(concepts that contain similar meaning but is expressed differently in
different modalities), and effectively leveraging unimodal representations in
cross-domain generative tasks, without compromising their original unimodal
capabilities.
  We propose Zipper, a multi-tower decoder architecture that addresses these
concerns by using cross-attention to flexibly compose multimodal generative
models from independently pre-trained unimodal decoders. In our experiments
fusing speech and text modalities, we show the proposed architecture performs
very competitively in scenarios with limited aligned text-speech data. We also
showcase the flexibility of our model to selectively maintain unimodal (e.g.,
text-to-text generation) generation performance by freezing the corresponding
modal tower (e.g. text). In cross-modal tasks such as automatic speech
recognition (ASR) where the output modality is text, we show that freezing the
text backbone results in negligible performance degradation. In cross-modal
tasks such as text-to-speech generation (TTS) where the output modality is
speech, we show that using a pre-trained speech backbone results in superior
performance to the baseline.
665888130a93bd5f0afe9a83
Vicky Zayats
665888130a93bd5f0afe9a84
Peter Chen
665888130a93bd5f0afe9a85
Melissa Merrari
665888130a93bd5f0afe9a86
Dirk Padfield
Zipper: A Multi-Tower Decoder Architecture for Fusing Modalities
Integrating multiple generative foundation models, especially those trained
on different modalities, into something greater than the sum of its parts poses
significant challenges. Two key hurdles are the availability of aligned data
(concepts that contain similar meaning but is expressed differently in
different modalities), and effectively leveraging unimodal representations in
cross-domain generative tasks, without compromising their original unimodal
capabilities.
  We propose Zipper, a multi-tower decoder architecture that addresses these
concerns by using cross-attention to flexibly compose multimodal generative
models from independently pre-trained unimodal decoders. In our experiments
fusing speech and text modalities, we show the proposed architecture performs
very competitively in scenarios with limited aligned text-speech data. We also
showcase the flexibility of our model to selectively maintain unimodal (e.g.,
text-to-text generation) generation performance by freezing the corresponding
modal tower (e.g. text). In cross-modal tasks such as automatic speech
recognition (ASR) where the output modality is text, we show that freezing the
text backbone results in negligible performance degradation. In cross-modal
tasks such as text-to-speech generation (TTS) where the output modality is
speech, we show that using a pre-trained speech backbone results in superior
performance to the baseline.
Zipper, a multi-tower decoder using cross-attention, integrates unimodal generative models for effective cross-modal tasks with limited aligned data while maintaining unimodal performance.
generative foundation models cross-attention multi-tower decoder aligned data unimodal representations cross-domain generative tasks text-to-text generation automatic speech recognition text-to-speech generation pre-trained speech backbone