What Matters in Training a GPT4-Style Language Model with Multimodal
  Inputs?
Recent advancements in Large Language Models (LLMs) such as GPT4 have
displayed exceptional multi-modal capabilities in following open-ended
instructions given images. However, the performance of these models heavily
relies on design choices such as network structures, training data, and
training strategies, and these choices have not been extensively discussed in
the literature, making it difficult to quantify progress in this field. To
address this issue, this paper presents a systematic and comprehensive study,
quantitatively and qualitatively, on training such models. We implement over 20
variants with controlled settings. Concretely, for network structures, we
compare different LLM backbones and model designs. For training data, we
investigate the impact of data and sampling strategies. For instructions, we
explore the influence of diversified prompts on the instruction-following
ability of the trained models. For benchmarks, we contribute the first, to our
best knowledge, comprehensive evaluation set including both image and video
tasks through crowd-sourcing. Based on our findings, we present Lynx, which
performs the most accurate multi-modal understanding while keeping the best
multi-modal generation ability compared to existing open-sourced GPT4-style
models.
64a62bf085638d0e698cf2d7
Yan Zeng
admin_assigned
2023-07-06T15:04:27.097Z
640ef2b2f2d7c41a1e9b6013
/avatars/556263c07df93942c22cda64e65b2e41.svg
Yan Zeng
user
zengyan-97
64a62bf085638d0e698cf2d8
Hanbo Zhang
64a62bf085638d0e698cf2d9
Jiani Zheng
claimed_verified
2023-11-21T15:00:53.949Z
64531f631a57e1179c203e6b
https://cdn-avatars.huggingface.co/v1/production/uploads/64531f631a57e1179c203e6b/C_J7pXFLqoJoHYPPhK3J9.jpeg
zjn
user
garlicisnotmyfavor
64a62bf085638d0e698cf2da
Jiangnan Xia
64a62bf085638d0e698cf2db
Guoqiang Wei
claimed_verified
2024-06-17T07:35:07.884Z
63160a1d29411a6864b0e585
https://cdn-avatars.huggingface.co/v1/production/uploads/1668083049513-63160a1d29411a6864b0e585.jpeg
Guoqiang Wei
user
weigq
64a62bf085638d0e698cf2dc
Yang Wei
64a62bf085638d0e698cf2dd
Yuchen Zhang
64a62bf085638d0e698cf2de
Tao Kong
What Matters in Training a GPT4-Style Language Model with Multimodal
  Inputs?
Recent advancements in Large Language Models (LLMs) such as GPT4 have
displayed exceptional multi-modal capabilities in following open-ended
instructions given images. However, the performance of these models heavily
relies on design choices such as network structures, training data, and
training strategies, and these choices have not been extensively discussed in
the literature, making it difficult to quantify progress in this field. To
address this issue, this paper presents a systematic and comprehensive study,
quantitatively and qualitatively, on training such models. We implement over 20
variants with controlled settings. Concretely, for network structures, we
compare different LLM backbones and model designs. For training data, we
investigate the impact of data and sampling strategies. For instructions, we
explore the influence of diversified prompts on the instruction-following
ability of the trained models. For benchmarks, we contribute the first, to our
best knowledge, comprehensive evaluation set including both image and video
tasks through crowd-sourcing. Based on our findings, we present Lynx, which
performs the most accurate multi-modal understanding while keeping the best
multi-modal generation ability compared to existing open-sourced GPT4-style
models.
A comprehensive study examines design choices impacting multi-modal capabilities of Large Language Models, introducing Lynx as the most accurate and versatile multi-modal model among existing GPT4-style models.
Large Language Models (LLMs) GPT4 multi-modal capabilities network structures LLM backbones model designs training data data and sampling strategies instruction-following ability comprehensive evaluation set multi-modal understanding multi-modal generation ability Lynx