List Items One by One: A New Data Source and Learning Paradigm for
  Multimodal LLMs
Set-of-Mark (SoM) Prompting unleashes the visual grounding capability of
GPT-4V, by enabling the model to associate visual objects with tags inserted on
the image. These tags, marked with alphanumerics, can be indexed via text
tokens for easy reference. Despite the extraordinary performance from GPT-4V,
we observe that other Multimodal Large Language Models (MLLMs) struggle to
understand these visual tags. To promote the learning of SoM prompting for
open-source models, we propose a new learning paradigm: "list items one by
one," which asks the model to enumerate and describe all visual tags placed on
the image following the alphanumeric orders of tags. By integrating our curated
dataset with other visual instruction tuning datasets, we are able to equip
existing MLLMs with the SoM prompting ability. Furthermore, we evaluate our
finetuned SoM models on five MLLM benchmarks. We find that this new dataset,
even in a relatively small size (10k-30k images with tags), significantly
enhances visual reasoning capabilities and reduces hallucinations for MLLMs.
Perhaps surprisingly, these improvements persist even when the visual tags are
omitted from input images during inference. This suggests the potential of
"list items one by one" as a new paradigm for training MLLMs, which strengthens
the object-text alignment through the use of visual tags in the training stage.
Finally, we conduct analyses by probing trained models to understand the
working mechanism of SoM. Our code and data are available at
https://github.com/zzxslp/SoM-LLaVA.
662b2d274149b1a3b0090f66
An Yan
claimed_verified
2024-04-28T19:30:08.776Z
634dfb9bbe5a827d48749f39
/avatars/a8df0eeb1df39f3935dd686a94768f30.svg
An Yan
user
zzxslp
662b2d274149b1a3b0090f67
Zhengyuan Yang
admin_assigned
2024-04-26T10:16:42.867Z
630713411801ecc7d2592a7c
/avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
Zhengyuan Yang
user
zyang39
662b2d274149b1a3b0090f68
Junda Wu
claimed_verified
2024-10-03T08:32:12.372Z
616c0cd640e2f69baa1c7adf
/avatars/9ad6ada73b3fb2d5fb696ec91f8edb1b.svg
Junda Wu
user
JoshuaW1997
662b2d274149b1a3b0090f69
Wanrong Zhu
admin_assigned
2024-04-26T10:17:31.249Z
633a7e91c8eb7b089034fb3c
/avatars/0e03d80aa469de131cc17bb0c5c016b9.svg
Wanrong Zhu
user
VegB
662b2d274149b1a3b0090f6a
Jianwei Yang
admin_assigned
2024-04-26T10:18:00.719Z
6125df7f25027fb1ea9c7a41
https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
Jianwei Yang
user
jw2yang
662b2d274149b1a3b0090f6b
Linjie Li
662b2d274149b1a3b0090f6c
Kevin Lin
admin_assigned
2024-04-26T10:18:42.974Z
6298fd95b58e71e2ac9f3ad8
/avatars/7d34644d537bc5c17cf1e4ce4095355c.svg
Kevin Lin
user
kevinlin311tw
662b2d274149b1a3b0090f6d
Jianfeng Wang
admin_assigned
2024-04-26T10:19:04.731Z
641404e74e5305c14f269b3c
/avatars/0dbe862cde7e4bef25bedf860584d8c1.svg
Jianfeng Wang
user
amsword8
662b2d274149b1a3b0090f6e
Julian McAuley
662b2d274149b1a3b0090f6f
Jianfeng Gao
admin_assigned
2024-04-26T10:19:16.215Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
662b2d274149b1a3b0090f70
Lijuan Wang
admin_assigned
2024-04-26T10:19:22.481Z
6413521d4e5305c14f22e110
/avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
Lijuan Wang
user
Lijuan
List Items One by One: A New Data Source and Learning Paradigm for
  Multimodal LLMs
Set-of-Mark (SoM) Prompting unleashes the visual grounding capability of
GPT-4V, by enabling the model to associate visual objects with tags inserted on
the image. These tags, marked with alphanumerics, can be indexed via text
tokens for easy reference. Despite the extraordinary performance from GPT-4V,
we observe that other Multimodal Large Language Models (MLLMs) struggle to
understand these visual tags. To promote the learning of SoM prompting for
open-source models, we propose a new learning paradigm: "list items one by
one," which asks the model to enumerate and describe all visual tags placed on
the image following the alphanumeric orders of tags. By integrating our curated
dataset with other visual instruction tuning datasets, we are able to equip
existing MLLMs with the SoM prompting ability. Furthermore, we evaluate our
finetuned SoM models on five MLLM benchmarks. We find that this new dataset,
even in a relatively small size (10k-30k images with tags), significantly
enhances visual reasoning capabilities and reduces hallucinations for MLLMs.
Perhaps surprisingly, these improvements persist even when the visual tags are
omitted from input images during inference. This suggests the potential of
"list items one by one" as a new paradigm for training MLLMs, which strengthens
the object-text alignment through the use of visual tags in the training stage.
Finally, we conduct analyses by probing trained models to understand the
working mechanism of SoM. Our code and data are available at
https://github.com/zzxslp/SoM-LLaVA.
A new paradigm, "list items one by one," enhances multimodal large language models' visual reasoning capability through the use of visual tags in training, even when these tags are omitted during inference.
GPT-4V Multimodal Large Language Models (MLLMs) SoM prompting list items one by one visual tags alphanumeric orders visual instruction tuning MLLM benchmarks visual reasoning capabilities hallucinations object-text alignment