PandaGPT: One Model To Instruction-Follow Them All
We present PandaGPT, an approach to emPower large lANguage moDels with visual
and Auditory instruction-following capabilities. Our pilot experiments show
that PandaGPT can perform complex tasks such as detailed image description
generation, writing stories inspired by videos, and answering questions about
audios. More interestingly, PandaGPT can take multimodal inputs simultaneously
and compose their semantics naturally. For example, PandaGPT can connect how
objects look in an image/video and how they sound in an audio. To do so,
PandaGPT combines the multimodal encoders from ImageBind and the large language
models from Vicuna. Notably, only aligned image-text pairs are required for the
training of PandaGPT. Thanks to the strong capability of ImageBind in embedding
data from different modalities into the same space, PandaGPT displays emergent,
i.e. zero-shot, cross-modal behaviors for data other than image and text (e.g.,
video, audio, depth, thermal, and IMU). We hope that PandaGPT serves as an
initial step toward building AGI that can perceive and understand inputs in
different modalities holistically, as we humans do. Our project page is at
https://panda-gpt.github.io/.
6474192abf9b32c6bbcd081a
Yixuan Su
claimed_verified
2023-05-30T14:15:00.810Z
6144c40da8f4f7baca305243
/avatars/1f9bbb70f859f34563fdd0a7de4cd254.svg
Yixuan Su
user
pangpang666
6474192abf9b32c6bbcd081b
Tian Lan
claimed_verified
2024-05-13T10:11:57.273Z
627ca439a09edfe62239c671
https://cdn-avatars.huggingface.co/v1/production/uploads/1652335660508-noauth.jpeg
Tian Lan
user
GMFTBY
6474192abf9b32c6bbcd081c
Huayang Li
claimed_verified
2023-05-30T14:15:43.811Z
64672e286192d3914221d633
/avatars/a26908ebcbb74feaf44fe83a44fef8ac.svg
Huayang
user
huayangli
6474192abf9b32c6bbcd081d
Jialu Xu
6474192abf9b32c6bbcd081e
Yan Wang
6474192abf9b32c6bbcd081f
Deng Cai
claimed_verified
2023-05-30T14:15:40.790Z
646735a1946476c5d214e5b2
/avatars/23392fda23fa6288fcb309f7eaa73472.svg
Deng Cai
user
jcyk
PandaGPT: One Model To Instruction-Follow Them All
We present PandaGPT, an approach to emPower large lANguage moDels with visual
and Auditory instruction-following capabilities. Our pilot experiments show
that PandaGPT can perform complex tasks such as detailed image description
generation, writing stories inspired by videos, and answering questions about
audios. More interestingly, PandaGPT can take multimodal inputs simultaneously
and compose their semantics naturally. For example, PandaGPT can connect how
objects look in an image/video and how they sound in an audio. To do so,
PandaGPT combines the multimodal encoders from ImageBind and the large language
models from Vicuna. Notably, only aligned image-text pairs are required for the
training of PandaGPT. Thanks to the strong capability of ImageBind in embedding
data from different modalities into the same space, PandaGPT displays emergent,
i.e. zero-shot, cross-modal behaviors for data other than image and text (e.g.,
video, audio, depth, thermal, and IMU). We hope that PandaGPT serves as an
initial step toward building AGI that can perceive and understand inputs in
different modalities holistically, as we humans do. Our project page is at
https://panda-gpt.github.io/.
PandaGPT integrates multimodal encoders and large language models to perform complex tasks involving images, videos, and audio, demonstrating zero-shot cross-modal understanding.
PandaGPT image-bind Vicuna multimodal encoders large language models image-text pairs zero-shot cross-modal behaviors