HEMM: Holistic Evaluation of Multimodal Foundation Models
Multimodal foundation models that can holistically process text alongside
images, video, audio, and other sensory modalities are increasingly used in a
variety of real-world applications. However, it is challenging to characterize
and study progress in multimodal foundation models, given the range of possible
modeling decisions, tasks, and domains. In this paper, we introduce Holistic
Evaluation of Multimodal Models (HEMM) to systematically evaluate the
capabilities of multimodal foundation models across a set of 3 dimensions:
basic skills, information flow, and real-world use cases. Basic multimodal
skills are internal abilities required to solve problems, such as learning
interactions across modalities, fine-grained alignment, multi-step reasoning,
and the ability to handle external knowledge. Information flow studies how
multimodal content changes during a task through querying, translation,
editing, and fusion. Use cases span domain-specific challenges introduced in
real-world multimedia, affective computing, natural sciences, healthcare, and
human-computer interaction applications. Through comprehensive experiments
across the 30 tasks in HEMM, we (1) identify key dataset dimensions (e.g.,
basic skills, information flows, and use cases) that pose challenges to today's
models, and (2) distill performance trends regarding how different modeling
dimensions (e.g., scale, pre-training data, multimodal alignment, pre-training,
and instruction tuning objectives) influence performance. Our conclusions
regarding challenging multimodal interactions, use cases, and tasks requiring
reasoning and external knowledge, the benefits of data and model scale, and the
impacts of instruction tuning yield actionable insights for future work in
multimodal foundation models.
668b8960ccbf926e742808b7
Paul Pu Liang
668b8960ccbf926e742808b8
Akshay Goindani
admin_assigned
2024-07-09T07:50:40.527Z
649de1288f9890dcc25bdf19
/avatars/caa7ccd06ccd49c5d14de317e21768d9.svg
Akshay Goindani
user
akshayg08
668b8960ccbf926e742808b9
Talha Chafekar
admin_assigned
2024-07-09T07:50:34.356Z
6146f91079d11bc3c1714775
/avatars/418247a90388a59f5d1ee8899144b329.svg
Talha Chafekar
user
talha1503
668b8960ccbf926e742808ba
Leena Mathur
admin_assigned
2024-07-09T07:50:27.878Z
62bb76d32b4d8cb1f9710f63
/avatars/ed695f5a39e1e6a2b2207ca0afad9eb9.svg
Leena Mathur
user
lmathur
668b8960ccbf926e742808bb
Haofei Yu
admin_assigned
2024-07-09T07:50:17.467Z
636453547cf2c0b4f0a3ee1e
/avatars/a453340b44d08eec2281ecbe5e993707.svg
Haofei Yu
user
lwaekfjlk
668b8960ccbf926e742808bc
Ruslan Salakhutdinov
668b8960ccbf926e742808bd
Louis-Philippe Morency
HEMM: Holistic Evaluation of Multimodal Foundation Models
Multimodal foundation models that can holistically process text alongside
images, video, audio, and other sensory modalities are increasingly used in a
variety of real-world applications. However, it is challenging to characterize
and study progress in multimodal foundation models, given the range of possible
modeling decisions, tasks, and domains. In this paper, we introduce Holistic
Evaluation of Multimodal Models (HEMM) to systematically evaluate the
capabilities of multimodal foundation models across a set of 3 dimensions:
basic skills, information flow, and real-world use cases. Basic multimodal
skills are internal abilities required to solve problems, such as learning
interactions across modalities, fine-grained alignment, multi-step reasoning,
and the ability to handle external knowledge. Information flow studies how
multimodal content changes during a task through querying, translation,
editing, and fusion. Use cases span domain-specific challenges introduced in
real-world multimedia, affective computing, natural sciences, healthcare, and
human-computer interaction applications. Through comprehensive experiments
across the 30 tasks in HEMM, we (1) identify key dataset dimensions (e.g.,
basic skills, information flows, and use cases) that pose challenges to today's
models, and (2) distill performance trends regarding how different modeling
dimensions (e.g., scale, pre-training data, multimodal alignment, pre-training,
and instruction tuning objectives) influence performance. Our conclusions
regarding challenging multimodal interactions, use cases, and tasks requiring
reasoning and external knowledge, the benefits of data and model scale, and the
impacts of instruction tuning yield actionable insights for future work in
multimodal foundation models.
Holistic Evaluation of Multimodal Models (HEMM) systematically assesses multimodal foundation models across basic skills, information flow, and real-world use cases, providing insights into model capabilities and challenges.
Holistic Evaluation of Multimodal Models HEMM multimodal foundation models basic multimodal skills inter-modality learning fine-grained alignment multi-step reasoning external knowledge information flow multimodal content querying translation editing fusion real-world use cases multimedia affective computing natural sciences healthcare human-computer interaction