DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via
  Multi-Modal Causal Attention
Most of the existing multi-modal models, hindered by their incapacity to
adeptly manage interleaved image-and-text inputs in multi-image, multi-round
dialogues, face substantial constraints in resource allocation for training and
data accessibility, impacting their adaptability and scalability across varied
interaction realms. To address this, we present the DeepSpeed-VisualChat
framework, designed to optimize Large Language Models (LLMs) by incorporating
multi-modal capabilities, with a focus on enhancing the proficiency of Large
Vision and Language Models in handling interleaved inputs. Our framework is
notable for (1) its open-source support for multi-round and multi-image
dialogues, (2) introducing an innovative multi-modal causal attention
mechanism, and (3) utilizing data blending techniques on existing datasets to
assure seamless interactions in multi-round, multi-image conversations.
Compared to existing frameworks, DeepSpeed-VisualChat shows superior
scalability up to 70B parameter language model size, representing a significant
advancement in multi-modal language models and setting a solid foundation for
future explorations.
65127f81e7fb861a6aa86ee1
Zhewei Yao
admin_assigned
2023-09-26T13:26:14.275Z
6352e1953199c2612ec2ff01
/avatars/f75444f1b7cba63ed085774a950a3fc7.svg
Zhewei Yao
user
zheweiyao
65127f81e7fb861a6aa86ee2
Xiaoxia Wu
claimed_verified
2023-09-26T18:51:25.710Z
619721da1726b360d54d9ff7
/avatars/f5f8bf0f372bef58848530dd7d7f58de.svg
wu
user
xiaoxia-microsoft
65127f81e7fb861a6aa86ee3
Conglong Li
admin_assigned
2023-09-26T13:40:23.730Z
617adc6d941993035fbaf2b3
/avatars/8de7dc3a6e202a49ac9c061e93d0fc92.svg
Conglong Li
user
conglongli
65127f81e7fb861a6aa86ee4
Minjia Zhang
admin_assigned
2023-09-26T13:40:39.286Z
6305b5e39d2531fabd195c5f
https://cdn-avatars.huggingface.co/v1/production/uploads/6305b5e39d2531fabd195c5f/oGS5leXVrB8Ct_BF96VkR.jpeg
Zhang
user
Minjia
65127f81e7fb861a6aa86ee5
Heyang Qi
admin_assigned
2023-09-26T13:40:46.337Z
63bc6a611374e3ef9134a24e
/avatars/9df037b9cf7d7952a10d919e24a2ffa2.svg
Heyang Qin
user
heyangqin
65127f81e7fb861a6aa86ee6
Olatunji Ruwase
admin_assigned
2023-09-26T13:40:53.473Z
6499d685ea2cdac80992e742
/avatars/c32741e7fc57c9a08722fab3877a7b81.svg
Olatunji Ruwase
user
tjruwase
65127f81e7fb861a6aa86ee7
Ammar Ahmad Awan
admin_assigned
2023-09-26T13:41:09.098Z
6324b8417d6d0cbbe2c65019
/avatars/7c1a468daaa984203a600a3b3526298f.svg
Ammar Awan
user
ammarawan
65127f81e7fb861a6aa86ee8
Samyam Rajbhandari
admin_assigned
2023-09-26T13:41:16.702Z
63238e79d444e1d90888ea03
/avatars/cd0accb5ddec4f2ae23c94da4094b5e5.svg
Samyam Rajbhandari
user
samyam
65127f81e7fb861a6aa86ee9
Yuxiong He
DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via
  Multi-Modal Causal Attention
Most of the existing multi-modal models, hindered by their incapacity to
adeptly manage interleaved image-and-text inputs in multi-image, multi-round
dialogues, face substantial constraints in resource allocation for training and
data accessibility, impacting their adaptability and scalability across varied
interaction realms. To address this, we present the DeepSpeed-VisualChat
framework, designed to optimize Large Language Models (LLMs) by incorporating
multi-modal capabilities, with a focus on enhancing the proficiency of Large
Vision and Language Models in handling interleaved inputs. Our framework is
notable for (1) its open-source support for multi-round and multi-image
dialogues, (2) introducing an innovative multi-modal causal attention
mechanism, and (3) utilizing data blending techniques on existing datasets to
assure seamless interactions in multi-round, multi-image conversations.
Compared to existing frameworks, DeepSpeed-VisualChat shows superior
scalability up to 70B parameter language model size, representing a significant
advancement in multi-modal language models and setting a solid foundation for
future explorations.
A framework called DeepSpeed-VisualChat optimizes LLMs for multi-modal, multi-image, and multi-round dialogues using an innovative causal attention mechanism and data blending techniques.
DeepSpeed-VisualChat Large Language Models (LLMs) Large Vision and Language Models multi-modal causal attention mechanism data blending techniques