X-LLM: Bootstrapping Advanced Large Language Models by Treating
  Multi-Modalities as Foreign Languages
Large language models (LLMs) have demonstrated remarkable language abilities.
GPT-4, based on advanced LLMs, exhibits extraordinary multimodal capabilities
beyond previous visual language models. We attribute this to the use of more
advanced LLMs compared with previous multimodal models. Unfortunately, the
model architecture and training strategies of GPT-4 are unknown. To endow LLMs
with multimodal capabilities, we propose X-LLM, which converts Multi-modalities
(images, speech, videos) into foreign languages using X2L interfaces and inputs
them into a large Language model (ChatGLM). Specifically, X-LLM aligns multiple
frozen single-modal encoders and a frozen LLM using X2L interfaces, where ``X''
denotes multi-modalities such as image, speech, and videos, and ``L'' denotes
languages. X-LLM's training consists of three stages: (1) Converting Multimodal
Information: The first stage trains each X2L interface to align with its
respective single-modal encoder separately to convert multimodal information
into languages. (2) Aligning X2L representations with the LLM: single-modal
encoders are aligned with the LLM through X2L interfaces independently. (3)
Integrating multiple modalities: all single-modal encoders are aligned with the
LLM through X2L interfaces to integrate multimodal capabilities into the LLM.
Our experiments show that X-LLM demonstrates impressive multimodel chat
abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen
images/instructions, and yields a 84.5\% relative score compared with GPT-4 on
a synthetic multimodal instruction-following dataset. And we also conduct
quantitative tests on using LLM for ASR and multimodal ASR, hoping to promote
the era of LLM-based speech recognition.
64599eba4fe72fae522b3322
Feilong Chen
admin_assigned
2023-06-02T14:19:38.048Z
64127917227028b63e31fec8
/avatars/23d7b088a5c8bdf8fb2e173216ffe316.svg
Feilong Chen
user
phellonchen
64599eba4fe72fae522b3323
Minglun Han
claimed_verified
2023-05-09T08:52:33.300Z
62fa3fdb0697d224219b38fa
/avatars/38297ed87c87e1dd58feedaffc355218.svg
mlhan
user
mlhan
64599eba4fe72fae522b3324
Haozhi Zhao
64599eba4fe72fae522b3325
Qingyang Zhang
64599eba4fe72fae522b3326
Jing Shi
64599eba4fe72fae522b3327
Shuang Xu
64599eba4fe72fae522b3328
Bo Xu
X-LLM: Bootstrapping Advanced Large Language Models by Treating
  Multi-Modalities as Foreign Languages
Large language models (LLMs) have demonstrated remarkable language abilities.
GPT-4, based on advanced LLMs, exhibits extraordinary multimodal capabilities
beyond previous visual language models. We attribute this to the use of more
advanced LLMs compared with previous multimodal models. Unfortunately, the
model architecture and training strategies of GPT-4 are unknown. To endow LLMs
with multimodal capabilities, we propose X-LLM, which converts Multi-modalities
(images, speech, videos) into foreign languages using X2L interfaces and inputs
them into a large Language model (ChatGLM). Specifically, X-LLM aligns multiple
frozen single-modal encoders and a frozen LLM using X2L interfaces, where ``X''
denotes multi-modalities such as image, speech, and videos, and ``L'' denotes
languages. X-LLM's training consists of three stages: (1) Converting Multimodal
Information: The first stage trains each X2L interface to align with its
respective single-modal encoder separately to convert multimodal information
into languages. (2) Aligning X2L representations with the LLM: single-modal
encoders are aligned with the LLM through X2L interfaces independently. (3)
Integrating multiple modalities: all single-modal encoders are aligned with the
LLM through X2L interfaces to integrate multimodal capabilities into the LLM.
Our experiments show that X-LLM demonstrates impressive multimodel chat
abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen
images/instructions, and yields a 84.5\% relative score compared with GPT-4 on
a synthetic multimodal instruction-following dataset. And we also conduct
quantitative tests on using LLM for ASR and multimodal ASR, hoping to promote
the era of LLM-based speech recognition.
X-LLM integrates multimodal information into language models using X2L interfaces, demonstrating multimodal chat abilities competitive with GPT-4.
LLMs GPT-4 multimodal capabilities X-LLM X2L interfaces single-modal encoders language models multimodal chat multimodal instruction-following dataset ASR multimodal ASR