Multi-Dimensional Insights: Benchmarking Real-World Personalization in
  Large Multimodal Models
The rapidly developing field of large multimodal models (LMMs) has led to the
emergence of diverse models with remarkable capabilities. However, existing
benchmarks fail to comprehensively, objectively and accurately evaluate whether
LMMs align with the diverse needs of humans in real-world scenarios. To bridge
this gap, we propose the Multi-Dimensional Insights (MDI) benchmark, which
includes over 500 images covering six common scenarios of human life. Notably,
the MDI-Benchmark offers two significant advantages over existing evaluations:
(1) Each image is accompanied by two types of questions: simple questions to
assess the model's understanding of the image, and complex questions to
evaluate the model's ability to analyze and reason beyond basic content. (2)
Recognizing that people of different age groups have varying needs and
perspectives when faced with the same scenario, our benchmark stratifies
questions into three age categories: young people, middle-aged people, and
older people. This design allows for a detailed assessment of LMMs'
capabilities in meeting the preferences and needs of different age groups. With
MDI-Benchmark, the strong model like GPT-4o achieve 79% accuracy on age-related
tasks, indicating that existing LMMs still have considerable room for
improvement in addressing real-world applications. Looking ahead, we anticipate
that the MDI-Benchmark will open new pathways for aligning real-world
personalization in LMMs. The MDI-Benchmark data and evaluation code are
available at https://mdi-benchmark.github.io/
676240aeb57a82f47d1a81f2
YiFan Zhang
676240aeb57a82f47d1a81f3
Shanglin Lei
676240aeb57a82f47d1a81f4
Runqi Qiao
claimed_verified
2024-12-19T18:03:40.564Z
6683a05e74fb1736a4b7c934
https://cdn-avatars.huggingface.co/v1/production/uploads/6683a05e74fb1736a4b7c934/eiz6qlqIUjAWGy5zfg8Cs.jpeg
QRQ
user
RichardQRQ
676240aeb57a82f47d1a81f5
Zhuoma GongQue
claimed_verified
2024-12-20T08:33:43.390Z
6762b973686e574cab4717ec
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/icacS5LkbX2S46b3Dqbni.png
zhuoma
user
Ethereal1l
676240aeb57a82f47d1a81f6
Xiaoshuai Song
676240aeb57a82f47d1a81f7
Guanting Dong
admin_assigned
2024-12-18T12:53:40.027Z
61cd4b833dd34ba1985e0753
https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
KABI
user
dongguanting
676240aeb57a82f47d1a81f8
Qiuna Tan
claimed_verified
2024-12-18T14:32:33.953Z
66615dd1c8bbe034b4b82029
/avatars/7ef1ddf2fa356efbeb2b3aa12284a1f0.svg
Zhang YiFan
user
zhangyifan666
676240aeb57a82f47d1a81f9
Zhe Wei
676240aeb57a82f47d1a81fa
Peiqing Yang
676240aeb57a82f47d1a81fb
Ye Tian
676240aeb57a82f47d1a81fc
Yadong Xue
admin_assigned
2024-12-18T12:52:43.539Z
64d4b5808b65d477e68f2fba
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/upUtVjLXZ9lAulngeFh_i.jpeg
Xue Yadong
user
ataraxy3
676240aeb57a82f47d1a81fd
Xiaofei Wang
676240aeb57a82f47d1a81fe
Honggang Zhang
Multi-Dimensional Insights: Benchmarking Real-World Personalization in
  Large Multimodal Models
The rapidly developing field of large multimodal models (LMMs) has led to the
emergence of diverse models with remarkable capabilities. However, existing
benchmarks fail to comprehensively, objectively and accurately evaluate whether
LMMs align with the diverse needs of humans in real-world scenarios. To bridge
this gap, we propose the Multi-Dimensional Insights (MDI) benchmark, which
includes over 500 images covering six common scenarios of human life. Notably,
the MDI-Benchmark offers two significant advantages over existing evaluations:
(1) Each image is accompanied by two types of questions: simple questions to
assess the model's understanding of the image, and complex questions to
evaluate the model's ability to analyze and reason beyond basic content. (2)
Recognizing that people of different age groups have varying needs and
perspectives when faced with the same scenario, our benchmark stratifies
questions into three age categories: young people, middle-aged people, and
older people. This design allows for a detailed assessment of LMMs'
capabilities in meeting the preferences and needs of different age groups. With
MDI-Benchmark, the strong model like GPT-4o achieve 79% accuracy on age-related
tasks, indicating that existing LMMs still have considerable room for
improvement in addressing real-world applications. Looking ahead, we anticipate
that the MDI-Benchmark will open new pathways for aligning real-world
personalization in LMMs. The MDI-Benchmark data and evaluation code are
available at https://mdi-benchmark.github.io/
The Multi-Dimensional Insights (MDI) benchmark evaluates large multimodal models across different scenarios and age groups to assess their real-world applicability, revealing areas for improvement in personalization.
large multimodal models LMMs Multi-Dimensional Insights MDI benchmark complex questions age categories GPT-4o