Two Giraffes in a Dirt Field: Using Game Play to Investigate Situation
  Modelling in Large Multimodal Models
While the situation has improved for text-only models, it again seems to be
the case currently that multimodal (text and image) models develop faster than
ways to evaluate them. In this paper, we bring a recently developed evaluation
paradigm from text models to multimodal models, namely evaluation through the
goal-oriented game (self) play, complementing reference-based and
preference-based evaluation. Specifically, we define games that challenge a
model's capability to represent a situation from visual information and align
such representations through dialogue. We find that the largest closed models
perform rather well on the games that we define, while even the best
open-weight models struggle with them. On further analysis, we find that the
exceptional deep captioning capabilities of the largest models drive some of
the performance. There is still room to grow for both kinds of models, ensuring
the continued relevance of the benchmark.
66790802f804642cd62b145b
Sherzod Hakimov
claimed_verified
2024-06-24T13:10:33.090Z
647061e39ad4008a29061769
https://cdn-avatars.huggingface.co/v1/production/uploads/647061e39ad4008a29061769/nHm9LquGI0Nh-MutiBvah.jpeg
Sherzod Hakimov
user
sherzod-hakimov
66790802f804642cd62b145c
Yerkezhan Abdullayeva
admin_assigned
2024-06-24T14:00:02.020Z
654cc79d34188e42e12ef229
/avatars/226d37b867f8a71a542e5e168bb78b42.svg
Yerkezhan Abdullayeva
user
yerkezhan
66790802f804642cd62b145d
Kushal Koshti
claimed_verified
2024-10-15T13:39:48.966Z
64fb4864c7f04f7cee779241
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/q-U-X_XvsnmuyXq0f1bH7.png
Koshti
user
Koshti10
66790802f804642cd62b145e
Antonia Schmidt
admin_assigned
2024-06-24T14:00:18.260Z
652401544f3337c6b1107b44
/avatars/d15e7702479fb9e7d8110905543ce44f.svg
Antonia Schmidt
user
AntoniaSch
66790802f804642cd62b145f
Yan Weiser
66790802f804642cd62b1460
Anne Beyer
admin_assigned
2024-06-24T14:00:32.483Z
62f529fd2f63f904a0c90788
https://cdn-avatars.huggingface.co/v1/production/uploads/62f529fd2f63f904a0c90788/rBpfRUwXVPe3_MJg5EhHO.jpeg
Anne Beyer
user
annebeyer
66790802f804642cd62b1461
David Schlangen
admin_assigned
2024-06-24T14:00:38.271Z
60c117aaea206f28132e3457
/avatars/94d1dab7d7c26f69b2f865bebd13f6ee.svg
David Schlangen
user
davids
Two Giraffes in a Dirt Field: Using Game Play to Investigate Situation
  Modelling in Large Multimodal Models
While the situation has improved for text-only models, it again seems to be
the case currently that multimodal (text and image) models develop faster than
ways to evaluate them. In this paper, we bring a recently developed evaluation
paradigm from text models to multimodal models, namely evaluation through the
goal-oriented game (self) play, complementing reference-based and
preference-based evaluation. Specifically, we define games that challenge a
model's capability to represent a situation from visual information and align
such representations through dialogue. We find that the largest closed models
perform rather well on the games that we define, while even the best
open-weight models struggle with them. On further analysis, we find that the
exceptional deep captioning capabilities of the largest models drive some of
the performance. There is still room to grow for both kinds of models, ensuring
the continued relevance of the benchmark.
Evaluation through goal-oriented self-play is applied to multimodal models, highlighting differences in performance compared to text-only models and revealing the influence of deep captioning capabilities.
multimodal models text models evaluation through the goal-oriented game self-play visual information dialogue deep captioning capabilities