Many-Shot In-Context Learning in Multimodal Foundation Models
Large language models are well-known to be effective at few-shot in-context
learning (ICL). Recent advancements in multimodal foundation models have
enabled unprecedentedly long context windows, presenting an opportunity to
explore their capability to perform ICL with many more demonstrating examples.
In this work, we evaluate the performance of multimodal foundation models
scaling from few-shot to many-shot ICL. We benchmark GPT-4o and Gemini 1.5 Pro
across 10 datasets spanning multiple domains (natural imagery, medical imagery,
remote sensing, and molecular imagery) and tasks (multi-class, multi-label, and
fine-grained classification). We observe that many-shot ICL, including up to
almost 2,000 multimodal demonstrating examples, leads to substantial
improvements compared to few-shot (<100 examples) ICL across all of the
datasets. Further, Gemini 1.5 Pro performance continues to improve log-linearly
up to the maximum number of tested examples on many datasets. Given the high
inference costs associated with the long prompts required for many-shot ICL, we
also explore the impact of batching multiple queries in a single API call. We
show that batching up to 50 queries can lead to performance improvements under
zero-shot and many-shot ICL, with substantial gains in the zero-shot setting on
multiple datasets, while drastically reducing per-query cost and latency.
Finally, we measure ICL data efficiency of the models, or the rate at which the
models learn from more demonstrating examples. We find that while GPT-4o and
Gemini 1.5 Pro achieve similar zero-shot performance across the datasets,
Gemini 1.5 Pro exhibits higher ICL data efficiency than GPT-4o on most
datasets. Our results suggest that many-shot ICL could enable users to
efficiently adapt multimodal foundation models to new applications and domains.
Our codebase is publicly available at
https://github.com/stanfordmlgroup/ManyICL .
6646ba2032fbd804f34d68aa
Yixing Jiang
claimed_verified
2024-05-20T07:38:33.700Z
64bbcbdde10455688fde3b51
/avatars/d5be06d419a1a6f3e53706c8e44cd7bc.svg
Yixing Jiang
user
jyx-su
6646ba2032fbd804f34d68ab
Jeremy Irvin
admin_assigned
2024-05-17T09:31:08.730Z
632cb34a4a4991e71155d4a6
/avatars/d87701cf30f425e018a7f89cb2d7cc5c.svg
Jeremy Irvin
user
jirvin16
6646ba2032fbd804f34d68ac
Ji Hun Wang
admin_assigned
2024-05-17T09:31:14.499Z
638003b79f1f158b014b4550
/avatars/9c268b96e22b6d01b2fbf805e6990d56.svg
Ji Hun Wang
user
JiHunWang
6646ba2032fbd804f34d68ad
Muhammad Ahmed Chaudhry
admin_assigned
2024-05-17T09:31:20.896Z
65ca76cd8e0d94e487338d97
/avatars/6853fa7ba90a8e727cff940eb5272a2c.svg
Muhammad Ahmed Chaudhry
user
MAhmedCh
6646ba2032fbd804f34d68ae
Jonathan H. Chen
6646ba2032fbd804f34d68af
Andrew Y. Ng
Many-Shot In-Context Learning in Multimodal Foundation Models
Large language models are well-known to be effective at few-shot in-context
learning (ICL). Recent advancements in multimodal foundation models have
enabled unprecedentedly long context windows, presenting an opportunity to
explore their capability to perform ICL with many more demonstrating examples.
In this work, we evaluate the performance of multimodal foundation models
scaling from few-shot to many-shot ICL. We benchmark GPT-4o and Gemini 1.5 Pro
across 10 datasets spanning multiple domains (natural imagery, medical imagery,
remote sensing, and molecular imagery) and tasks (multi-class, multi-label, and
fine-grained classification). We observe that many-shot ICL, including up to
almost 2,000 multimodal demonstrating examples, leads to substantial
improvements compared to few-shot (<100 examples) ICL across all of the
datasets. Further, Gemini 1.5 Pro performance continues to improve log-linearly
up to the maximum number of tested examples on many datasets. Given the high
inference costs associated with the long prompts required for many-shot ICL, we
also explore the impact of batching multiple queries in a single API call. We
show that batching up to 50 queries can lead to performance improvements under
zero-shot and many-shot ICL, with substantial gains in the zero-shot setting on
multiple datasets, while drastically reducing per-query cost and latency.
Finally, we measure ICL data efficiency of the models, or the rate at which the
models learn from more demonstrating examples. We find that while GPT-4o and
Gemini 1.5 Pro achieve similar zero-shot performance across the datasets,
Gemini 1.5 Pro exhibits higher ICL data efficiency than GPT-4o on most
datasets. Our results suggest that many-shot ICL could enable users to
efficiently adapt multimodal foundation models to new applications and domains.
Our codebase is publicly available at
https://github.com/stanfordmlgroup/ManyICL .
Multimodal foundation models exhibit significant performance improvements with many-shot in-context learning compared to few-shot, with Gemini 1.5 Pro showing higher data efficiency and better batch processing capabilities across various domains.
few-shot in-context learning GPT-4o Gemini 1.5 Pro multimodal foundation models in-context learning data efficiency batch processing