A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise
The surge of interest towards Multi-modal Large Language Models (MLLMs),
e.g., GPT-4V(ision) from OpenAI, has marked a significant trend in both
academia and industry. They endow Large Language Models (LLMs) with powerful
capabilities in visual understanding, enabling them to tackle diverse
multi-modal tasks. Very recently, Google released Gemini, its newest and most
capable MLLM built from the ground up for multi-modality. In light of the
superior reasoning capabilities, can Gemini challenge GPT-4V's leading position
in multi-modal learning? In this paper, we present a preliminary exploration of
Gemini Pro's visual understanding proficiency, which comprehensively covers
four domains: fundamental perception, advanced cognition, challenging vision
tasks, and various expert capacities. We compare Gemini Pro with the
state-of-the-art GPT-4V to evaluate its upper limits, along with the latest
open-sourced MLLM, Sphinx, which reveals the gap between manual efforts and
black-box systems. The qualitative samples indicate that, while GPT-4V and
Gemini showcase different answering styles and preferences, they can exhibit
comparable visual reasoning capabilities, and Sphinx still trails behind them
concerning domain generalizability. Specifically, GPT-4V tends to elaborate
detailed explanations and intermediate steps, and Gemini prefers to output a
direct and concise answer. The quantitative evaluation on the popular MME
benchmark also demonstrates the potential of Gemini to be a strong challenger
to GPT-4V. Our early investigation of Gemini also observes some common issues
of MLLMs, indicating that there still remains a considerable distance towards
artificial general intelligence. Our project for tracking the progress of MLLM
is released at
https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models.
6582505d1c4454dde6d3a6aa
Chaoyou Fu
admin_assigned
2023-12-20T09:12:04.184Z
653b268cd1041ca9188954da
/avatars/2529a821580f5d87ee46867c5dc5b1e5.svg
BradyFU
user
BradyFU
6582505d1c4454dde6d3a6ab
Renrui Zhang
claimed_verified
2024-03-22T08:40:05.948Z
645b8b2687c79b6ec0bb3b7a
/avatars/00a9db32a42dc950112bf2593bb109cb.svg
Renrui
user
ZrrSkywalker
6582505d1c4454dde6d3a6ac
Haojia Lin
admin_assigned
2023-12-20T09:13:21.749Z
64ffd436d522560505a94b8e
/avatars/02d4faac40ac203cb5d635cfcb39780c.svg
Haojia Lin
user
linhaojia13
6582505d1c4454dde6d3a6ad
Zihan Wang
6582505d1c4454dde6d3a6ae
Timin Gao
6582505d1c4454dde6d3a6af
Yongdong Luo
6582505d1c4454dde6d3a6b0
Yubo Huang
6582505d1c4454dde6d3a6b1
Zhengye Zhang
6582505d1c4454dde6d3a6b2
Longtian Qiu
6582505d1c4454dde6d3a6b3
Gaoxiang Ye
6582505d1c4454dde6d3a6b4
Yunhang Shen
admin_assigned
2023-12-20T09:19:11.278Z
6483143902f98c3f05aff915
https://cdn-avatars.huggingface.co/v1/production/uploads/6483143902f98c3f05aff915/ZhWFFgrlRsQf4MXiInh5p.jpeg
沈云航 Yunhang Shen
user
shenyunhang
6582505d1c4454dde6d3a6b5
Mengdan Zhang
6582505d1c4454dde6d3a6b6
Peixian Chen
6582505d1c4454dde6d3a6b7
Sirui Zhao
admin_assigned
2023-12-20T09:18:50.406Z
653a55ddae6a4080ce3eb860
/avatars/44a62d1e0b72a42690d1658e5b42f3a4.svg
Sirui Zhao
user
SiruiZhao
6582505d1c4454dde6d3a6b8
Xiawu Zheng
6582505d1c4454dde6d3a6b9
Shaohui Lin
6582505d1c4454dde6d3a6ba
Deqiang Jiang
6582505d1c4454dde6d3a6bb
Di Yin
6582505d1c4454dde6d3a6bc
Peng Gao
6582505d1c4454dde6d3a6bd
Ke Li
claimed_verified
2025-01-09T20:02:12.226Z
63280915eeee4dd858083092
/avatars/78347af4af42527d53e88d9969c5c934.svg
Ke Li
user
tristanli
6582505d1c4454dde6d3a6be
Xing Sun
admin_assigned
2023-12-20T09:16:45.196Z
647401e50da364bd0d002f2a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vPuPn7EV092mLBOM2YZXd.png
XING SUN
user
tedsun
6582505d1c4454dde6d3a6bf
Rongrong Ji
A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise
The surge of interest towards Multi-modal Large Language Models (MLLMs),
e.g., GPT-4V(ision) from OpenAI, has marked a significant trend in both
academia and industry. They endow Large Language Models (LLMs) with powerful
capabilities in visual understanding, enabling them to tackle diverse
multi-modal tasks. Very recently, Google released Gemini, its newest and most
capable MLLM built from the ground up for multi-modality. In light of the
superior reasoning capabilities, can Gemini challenge GPT-4V's leading position
in multi-modal learning? In this paper, we present a preliminary exploration of
Gemini Pro's visual understanding proficiency, which comprehensively covers
four domains: fundamental perception, advanced cognition, challenging vision
tasks, and various expert capacities. We compare Gemini Pro with the
state-of-the-art GPT-4V to evaluate its upper limits, along with the latest
open-sourced MLLM, Sphinx, which reveals the gap between manual efforts and
black-box systems. The qualitative samples indicate that, while GPT-4V and
Gemini showcase different answering styles and preferences, they can exhibit
comparable visual reasoning capabilities, and Sphinx still trails behind them
concerning domain generalizability. Specifically, GPT-4V tends to elaborate
detailed explanations and intermediate steps, and Gemini prefers to output a
direct and concise answer. The quantitative evaluation on the popular MME
benchmark also demonstrates the potential of Gemini to be a strong challenger
to GPT-4V. Our early investigation of Gemini also observes some common issues
of MLLMs, indicating that there still remains a considerable distance towards
artificial general intelligence. Our project for tracking the progress of MLLM
is released at
https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models.
Preliminary evaluation of Gemini Pro's visual understanding shows it to be a strong contender to GPT-4V in multi-modal learning, though both models face challenges in achieving domain generalization compared to open-sourced systems.
Multi-modal Large Language Models MLLMs GPT-4V Gemini visual understanding fundamental perception advanced cognition challenging vision tasks expert capacities MME benchmark domain generalization artificial general intelligence