From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on
  Generalizability, Trustworthiness and Causality through Four Modalities
Multi-modal Large Language Models (MLLMs) have shown impressive abilities in
generating reasonable responses with respect to multi-modal contents. However,
there is still a wide gap between the performance of recent MLLM-based
applications and the expectation of the broad public, even though the most
powerful OpenAI's GPT-4 and Google's Gemini have been deployed. This paper
strives to enhance understanding of the gap through the lens of a qualitative
study on the generalizability, trustworthiness, and causal reasoning
capabilities of recent proprietary and open-source MLLMs across four
modalities: ie, text, code, image, and video, ultimately aiming to improve the
transparency of MLLMs. We believe these properties are several representative
factors that define the reliability of MLLMs, in supporting various downstream
applications. To be specific, we evaluate the closed-source GPT-4 and Gemini
and 6 open-source LLMs and MLLMs. Overall we evaluate 230 manually designed
cases, where the qualitative results are then summarized into 12 scores (ie, 4
modalities times 3 properties). In total, we uncover 14 empirical findings that
are useful to understand the capabilities and limitations of both proprietary
and open-source MLLMs, towards more reliable downstream multi-modal
applications.
65b70a43e2510e886d92d1c8
Chaochao Lu
65b70a43e2510e886d92d1c9
Chen Qian
65b70a43e2510e886d92d1ca
Guodong Zheng
65b70a43e2510e886d92d1cb
Hongxing Fan
admin_assigned
2024-01-29T09:39:09.407Z
64a96a375a69e2ca889abdff
/avatars/f288c66ace09d907f132a79a740a3701.svg
fanhongxing
user
fanhongxing
65b70a43e2510e886d92d1cc
Hongzhi Gao
65b70a43e2510e886d92d1cd
Jie Zhang
65b70a43e2510e886d92d1ce
Jing Shao
65b70a43e2510e886d92d1cf
Jingyi Deng
claimed_verified
2024-01-30T17:15:33.312Z
65b914e6b1e017206da47f7a
/avatars/f5bcf66715dd8b3fa94fc4b424e3477a.svg
Jingyi Deng
user
JyiDeng
65b70a43e2510e886d92d1d0
Jinlan Fu
admin_assigned
2024-01-29T09:54:47.621Z
618497ea8aaadc9253c2dfa9
/avatars/2eb3954a99f5aede6f31b8ae49b8c910.svg
Fu Jinlan
user
Jinlan
65b70a43e2510e886d92d1d1
Kexin Huang
admin_assigned
2024-01-29T09:57:06.761Z
64ccd0104726a3f833823095
https://cdn-avatars.huggingface.co/v1/production/uploads/64ccd0104726a3f833823095/Q38tL8aD49BbcHACyvaea.jpeg
Kexin Huang
user
CaasiHUANG
65b70a43e2510e886d92d1d2
Kunchang Li
admin_assigned
2024-01-29T09:58:39.415Z
61fb81006374891646732f37
https://cdn-avatars.huggingface.co/v1/production/uploads/1643872995181-61fb81006374891646732f37.jpeg
Kunchang Li
user
Andy1621
65b70a43e2510e886d92d1d3
Lijun Li
65b70a43e2510e886d92d1d4
Limin Wang
65b70a43e2510e886d92d1d5
Lu Sheng
claimed_verified
2024-02-22T22:28:17.747Z
65b722dbe02a17f0f8d1cc6b
/avatars/65f20601ef9b8ebfdddadd737f9153d6.svg
Lu Sheng
user
lsheng2024
65b70a43e2510e886d92d1d6
Meiqi Chen
65b70a43e2510e886d92d1d7
Ming Zhang
claimed_verified
2024-01-29T08:24:31.975Z
65b71c0582d38451342f7334
/avatars/e00248fb99604fa8f4f33f50584d376d.svg
Ming Zhang
user
konglongge
65b70a43e2510e886d92d1d8
Qibing Ren
65b70a43e2510e886d92d1d9
Sirui Chen
65b70a43e2510e886d92d1da
Tao Gui
65b70a43e2510e886d92d1db
Wanli Ouyang
65b70a43e2510e886d92d1dc
Yali Wang
65b70a43e2510e886d92d1dd
Yan Teng
65b70a43e2510e886d92d1de
Yaru Wang
65b70a43e2510e886d92d1df
Yi Wang
65b70a43e2510e886d92d1e0
Yinan He
claimed_verified
2024-03-18T08:37:27.410Z
62aafa49f29ff279b51f0182
https://cdn-avatars.huggingface.co/v1/production/uploads/62aafa49f29ff279b51f0182/rQx8QFQGOY2qIhqJ8zSRj.jpeg
yinanhe
user
ynhe
65b70a43e2510e886d92d1e1
Yingchun Wang
65b70a43e2510e886d92d1e2
Yixu Wang
65b70a43e2510e886d92d1e3
Yongting Zhang
65b70a43e2510e886d92d1e4
Yu Qiao
65b70a43e2510e886d92d1e5
Yujiong Shen
65b70a43e2510e886d92d1e6
Yurong Mou
65b70a43e2510e886d92d1e7
Yuxi Chen
65b70a43e2510e886d92d1e8
Zaibin Zhang
65b70a43e2510e886d92d1e9
Zhelun Shi
admin_assigned
2024-01-29T09:40:35.221Z
64f5d8b05619fe1fe06ed1a7
/avatars/61016a0533760dafacefe8ce6e344318.svg
Zhelun Shi
user
CoachXP
65b70a43e2510e886d92d1ea
Zhenfei Yin
claimed_verified
2024-01-29T08:24:34.338Z
64e314ad24809d7fa0f20fbc
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/bHE0w_hjDFvU-Aul0_E7g.jpeg
Zhenfei Yin
user
JeremyYin
65b70a43e2510e886d92d1eb
Zhipin Wang
From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on
  Generalizability, Trustworthiness and Causality through Four Modalities
Multi-modal Large Language Models (MLLMs) have shown impressive abilities in
generating reasonable responses with respect to multi-modal contents. However,
there is still a wide gap between the performance of recent MLLM-based
applications and the expectation of the broad public, even though the most
powerful OpenAI's GPT-4 and Google's Gemini have been deployed. This paper
strives to enhance understanding of the gap through the lens of a qualitative
study on the generalizability, trustworthiness, and causal reasoning
capabilities of recent proprietary and open-source MLLMs across four
modalities: ie, text, code, image, and video, ultimately aiming to improve the
transparency of MLLMs. We believe these properties are several representative
factors that define the reliability of MLLMs, in supporting various downstream
applications. To be specific, we evaluate the closed-source GPT-4 and Gemini
and 6 open-source LLMs and MLLMs. Overall we evaluate 230 manually designed
cases, where the qualitative results are then summarized into 12 scores (ie, 4
modalities times 3 properties). In total, we uncover 14 empirical findings that
are useful to understand the capabilities and limitations of both proprietary
and open-source MLLMs, towards more reliable downstream multi-modal
applications.
A qualitative study evaluates the generalizability, trustworthiness, and causal reasoning capabilities of MLLMs across text, code, image, and video modalities to understand their reliability for downstream applications.
Multi-modal Large Language Models MLLMs generalizability trustworthiness causal reasoning OpenAI's GPT-4 Google's Gemini qualitative study closed-source open-source empirical findings