NExT-GPT: Any-to-Any Multimodal LLM
While recently Multimodal Large Language Models (MM-LLMs) have made exciting
strides, they mostly fall prey to the limitation of only input-side multimodal
understanding, without the ability to produce content in multiple modalities.
As we humans always perceive the world and communicate with people through
various modalities, developing any-to-any MM-LLMs capable of accepting and
delivering content in any modality becomes essential to human-level AI. To fill
the gap, we present an end-to-end general-purpose any-to-any MM-LLM system,
NExT-GPT. We connect an LLM with multimodal adaptors and different diffusion
decoders, enabling NExT-GPT to perceive inputs and generate outputs in
arbitrary combinations of text, images, videos, and audio. By leveraging the
existing well-trained highly-performing encoders and decoders, NExT-GPT is
tuned with only a small amount of parameter (1%) of certain projection layers,
which not only benefits low-cost training and also facilitates convenient
expansion to more potential modalities. Moreover, we introduce a
modality-switching instruction tuning (MosIT) and manually curate a
high-quality dataset for MosIT, based on which NExT-GPT is empowered with
complex cross-modal semantic understanding and content generation. Overall, our
research showcases the promising possibility of building an AI agent capable of
modeling universal modalities, paving the way for more human-like AI research
in the community.
64ffd83052aec889f49b61ea
Shengqiong Wu
admin_assigned
2023-09-12T11:00:46.264Z
64c139d867eff857ea51caa8
/avatars/4b7b3f41c2e2cfa21dd43bbac6e081ae.svg
Shengqiong Wu
user
ChocoWu
64ffd83052aec889f49b61eb
Hao Fei
admin_assigned
2023-09-12T11:01:39.654Z
647773a1168cb428e00e9a8f
https://cdn-avatars.huggingface.co/v1/production/uploads/647773a1168cb428e00e9a8f/NiRR3ScY6Plzjibfwy1hC.jpeg
Hao Fei
user
scofield7419
64ffd83052aec889f49b61ec
Leigang Qu
admin_assigned
2023-09-12T11:01:47.187Z
641a6e324182690729c860b5
/avatars/4599451d9775bf1aa11ef1e7011cb05a.svg
Leigang Qu
user
leigangqu
64ffd83052aec889f49b61ed
Wei Ji
claimed_verified
2023-09-14T07:31:49.655Z
65025db534ddd20327363af6
/avatars/ef7982780d2bc544882ed32441c0df5d.svg
Wei Ji
user
away0523
64ffd83052aec889f49b61ee
Tat-Seng Chua
NExT-GPT: Any-to-Any Multimodal LLM
While recently Multimodal Large Language Models (MM-LLMs) have made exciting
strides, they mostly fall prey to the limitation of only input-side multimodal
understanding, without the ability to produce content in multiple modalities.
As we humans always perceive the world and communicate with people through
various modalities, developing any-to-any MM-LLMs capable of accepting and
delivering content in any modality becomes essential to human-level AI. To fill
the gap, we present an end-to-end general-purpose any-to-any MM-LLM system,
NExT-GPT. We connect an LLM with multimodal adaptors and different diffusion
decoders, enabling NExT-GPT to perceive inputs and generate outputs in
arbitrary combinations of text, images, videos, and audio. By leveraging the
existing well-trained highly-performing encoders and decoders, NExT-GPT is
tuned with only a small amount of parameter (1%) of certain projection layers,
which not only benefits low-cost training and also facilitates convenient
expansion to more potential modalities. Moreover, we introduce a
modality-switching instruction tuning (MosIT) and manually curate a
high-quality dataset for MosIT, based on which NExT-GPT is empowered with
complex cross-modal semantic understanding and content generation. Overall, our
research showcases the promising possibility of building an AI agent capable of
modeling universal modalities, paving the way for more human-like AI research
in the community.
NExT-GPT, an any-to-any Multimodal Large Language Model, combines LLMs with multimodal adaptors and diffusion decoders to generate content across various modalities, enhanced by modality-switching instruction tuning and a curated dataset.
Multimodal Large Language Models MM-LLMs LLMs multimodal adaptors diffusion decoders parameter tuning projection layers modality-switching instruction tuning MosIT universal modalities