Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision,
  Language, Audio, and Action
We present Unified-IO 2, the first autoregressive multimodal model that is
capable of understanding and generating image, text, audio, and action. To
unify different modalities, we tokenize inputs and outputs -- images, text,
audio, action, bounding boxes, etc., into a shared semantic space and then
process them with a single encoder-decoder transformer model. Since training
with such diverse modalities is challenging, we propose various architectural
improvements to stabilize model training. We train our model from scratch on a
large multimodal pre-training corpus from diverse sources with a multimodal
mixture of denoisers objective. To learn an expansive set of skills, such as
following multimodal instructions, we construct and finetune on an ensemble of
120 datasets with prompts and augmentations. With a single unified model,
Unified-IO 2 achieves state-of-the-art performance on the GRIT benchmark and
strong results in more than 35 benchmarks, including image generation and
understanding, natural language understanding, video and audio understanding,
and robotic manipulation. We release all our models to the research community.
658e40a016227c7a2dd07eb4
Jiasen Lu
658e40a016227c7a2dd07eb5
Christopher Clark
658e40a016227c7a2dd07eb6
Sangho Lee
claimed_verified
2024-12-04T09:48:16.034Z
664fc02d6f16bbd9a168654d
/avatars/9f726ede9973706bc198845464238452.svg
Sangho Lee
user
sanghol
658e40a016227c7a2dd07eb7
Zichen Zhang
658e40a016227c7a2dd07eb8
Savya Khosla
658e40a016227c7a2dd07eb9
Ryan Marten
claimed_verified
2024-07-16T20:17:32.793Z
626c182a030a6e7363b6fe0a
https://cdn-avatars.huggingface.co/v1/production/uploads/626c182a030a6e7363b6fe0a/NaMcIOPVgJ6JFt8fojZWU.jpeg
Ryan Marten
user
ryanmarten
658e40a016227c7a2dd07eba
Derek Hoiem
658e40a016227c7a2dd07ebb
Aniruddha Kembhavi
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision,
  Language, Audio, and Action
We present Unified-IO 2, the first autoregressive multimodal model that is
capable of understanding and generating image, text, audio, and action. To
unify different modalities, we tokenize inputs and outputs -- images, text,
audio, action, bounding boxes, etc., into a shared semantic space and then
process them with a single encoder-decoder transformer model. Since training
with such diverse modalities is challenging, we propose various architectural
improvements to stabilize model training. We train our model from scratch on a
large multimodal pre-training corpus from diverse sources with a multimodal
mixture of denoisers objective. To learn an expansive set of skills, such as
following multimodal instructions, we construct and finetune on an ensemble of
120 datasets with prompts and augmentations. With a single unified model,
Unified-IO 2 achieves state-of-the-art performance on the GRIT benchmark and
strong results in more than 35 benchmarks, including image generation and
understanding, natural language understanding, video and audio understanding,
and robotic manipulation. We release all our models to the research community.
A unified autoregressive multimodal model handles image, text, audio, and action by tokenizing inputs into a shared space and processing them with a transformer architecture, achieving state-of-the-art performance across various benchmarks.
autoregressive multimodal model encoder-decoder transformer multimodal mixture of denoisers prompts augmentations GRIT benchmark image generation natural language understanding video and audio understanding robotic manipulation