Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and
  Text Integration
Although instruction-tuned large language models (LLMs) have exhibited
remarkable capabilities across various NLP tasks, their effectiveness on other
data modalities beyond text has not been fully studied. In this work, we
propose Macaw-LLM, a novel multi-modal LLM that seamlessly integrates visual,
audio, and textual information. Macaw-LLM consists of three main components: a
modality module for encoding multi-modal data, a cognitive module for
harnessing pretrained LLMs, and an alignment module for harmonizing diverse
representations. Our novel alignment module seamlessly bridges multi-modal
features to textual features, simplifying the adaptation process from the
modality modules to the cognitive module. In addition, we construct a
large-scale multi-modal instruction dataset in terms of multi-turn dialogue,
including 69K image instances and 50K video instances. We have made our data,
code and model publicly available, which we hope can pave the way for future
research in multi-modal LLMs and expand the capabilities of LLMs to handle
diverse data modalities and address complex real-world scenarios.
648bd92a9f4df9495b8adcb7
Chenyang Lyu
claimed_verified
2025-04-23T08:41:07.726Z
6527d8b077bceabaab382a75
/avatars/69caacf9153dbf6a3796693a968b363f.svg
Chenyang Lyu
user
ChenyangLyu
648bd92a9f4df9495b8adcb8
Minghao Wu
extracted_confirmed
2023-06-16T03:44:40.608Z
62d4bf8c97ab9eb08762a975
/avatars/73c6228e317cf37b4e3c3e7a4b3d8ae8.svg
Minghao Wu
user
minghaowu
648bd92a9f4df9495b8adcb9
Longyue Wang
admin_assigned
2023-06-19T11:59:44.116Z
636b030c328133bdb3a523bc
https://cdn-avatars.huggingface.co/v1/production/uploads/636b030c328133bdb3a523bc/f-OdbqqHiywkxQF1KVCLp.jpeg
Longyue Wang
user
longyuewang
648bd92a9f4df9495b8adcba
Xinting Huang
648bd92a9f4df9495b8adcbb
Bingshuai Liu
admin_assigned
2023-06-19T11:59:35.636Z
6347f4968af48bc2fb5663f8
/avatars/3941abd3cfdb68212db6106e2ce521ed.svg
Bingshuai Liu
user
bsliu
648bd92a9f4df9495b8adcbc
Zefeng Du
648bd92a9f4df9495b8adcbd
Shuming Shi
admin_assigned
2023-06-19T12:00:32.800Z
62eb8e9eaf9bed70f5ad5a7e
/avatars/971763f3b724d63bc64b7d3599cfc753.svg
Shuming Shi
user
Shuming
648bd92a9f4df9495b8adcbe
Zhaopeng Tu
claimed_verified
2024-11-28T11:44:29.528Z
67485743561b1e6f9579389f
/avatars/8a4cc63bd7be388010bc329bb74582a1.svg
Zhaopeng Tu
user
zptu
Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and
  Text Integration
Although instruction-tuned large language models (LLMs) have exhibited
remarkable capabilities across various NLP tasks, their effectiveness on other
data modalities beyond text has not been fully studied. In this work, we
propose Macaw-LLM, a novel multi-modal LLM that seamlessly integrates visual,
audio, and textual information. Macaw-LLM consists of three main components: a
modality module for encoding multi-modal data, a cognitive module for
harnessing pretrained LLMs, and an alignment module for harmonizing diverse
representations. Our novel alignment module seamlessly bridges multi-modal
features to textual features, simplifying the adaptation process from the
modality modules to the cognitive module. In addition, we construct a
large-scale multi-modal instruction dataset in terms of multi-turn dialogue,
including 69K image instances and 50K video instances. We have made our data,
code and model publicly available, which we hope can pave the way for future
research in multi-modal LLMs and expand the capabilities of LLMs to handle
diverse data modalities and address complex real-world scenarios.
Macaw-LLM integrates multimodal data including visual, audio, and text by employing a modality module, cognitive module, and novel alignment module, enhancing LLM capabilities across diverse data types.
instruction-tuned large language models multi-modal LLM modality module cognitive module alignment module multi-turn dialogue image instances video instances