Link-Context Learning for Multimodal LLMs
The ability to learn from context with novel concepts, and deliver
appropriate responses are essential in human conversations. Despite current
Multimodal Large Language Models (MLLMs) and Large Language Models (LLMs) being
trained on mega-scale datasets, recognizing unseen images or understanding
novel concepts in a training-free manner remains a challenge. In-Context
Learning (ICL) explores training-free few-shot learning, where models are
encouraged to ``learn to learn" from limited tasks and generalize to unseen
tasks. In this work, we propose link-context learning (LCL), which emphasizes
"reasoning from cause and effect" to augment the learning capabilities of
MLLMs. LCL goes beyond traditional ICL by explicitly strengthening the causal
relationship between the support set and the query set. By providing
demonstrations with causal links, LCL guides the model to discern not only the
analogy but also the underlying causal associations between data points, which
empowers MLLMs to recognize unseen images and understand novel concepts more
effectively. To facilitate the evaluation of this novel approach, we introduce
the ISEKAI dataset, comprising exclusively of unseen generated image-label
pairs designed for link-context learning. Extensive experiments show that our
LCL-MLLM exhibits strong link-context learning capabilities to novel concepts
over vanilla MLLMs. Code and data will be released at
https://github.com/isekai-portal/Link-Context-Learning.
64dc8a4698d50ccac684c74b
Yan Tai
claimed_verified
2023-08-17T13:53:27.017Z
647857b6c43296134a434fc6
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Cs9p9xSXFlP3Y24JNV7Am.jpeg
taiyan
user
Amoik
64dc8a4698d50ccac684c74c
Weichen Fan
claimed_verified
2023-08-17T15:03:55.077Z
6481764e8af4675862efb22e
/avatars/fc2e076bc861693f598a528a068a696e.svg
weichenfan
user
weepiess2383
64dc8a4698d50ccac684c74d
Zhao Zhang
claimed_verified
2024-04-24T11:14:03.803Z
62bc1adacaf01b9bec398547
https://cdn-avatars.huggingface.co/v1/production/uploads/1656494729797-noauth.png
Zhao Zhang
user
zbrl
64dc8a4698d50ccac684c74e
Feng Zhu
64dc8a4698d50ccac684c74f
Rui Zhao
admin_assigned
2023-08-16T14:41:22.403Z
6405d7cca577649430bcceb9
/avatars/befc09d3bc82685c8e205dcadfec13c0.svg
Rui Zhao
user
Robert0812
64dc8a4698d50ccac684c750
Ziwei Liu
admin_assigned
2023-08-16T14:38:40.872Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
Link-Context Learning for Multimodal LLMs
The ability to learn from context with novel concepts, and deliver
appropriate responses are essential in human conversations. Despite current
Multimodal Large Language Models (MLLMs) and Large Language Models (LLMs) being
trained on mega-scale datasets, recognizing unseen images or understanding
novel concepts in a training-free manner remains a challenge. In-Context
Learning (ICL) explores training-free few-shot learning, where models are
encouraged to ``learn to learn" from limited tasks and generalize to unseen
tasks. In this work, we propose link-context learning (LCL), which emphasizes
"reasoning from cause and effect" to augment the learning capabilities of
MLLMs. LCL goes beyond traditional ICL by explicitly strengthening the causal
relationship between the support set and the query set. By providing
demonstrations with causal links, LCL guides the model to discern not only the
analogy but also the underlying causal associations between data points, which
empowers MLLMs to recognize unseen images and understand novel concepts more
effectively. To facilitate the evaluation of this novel approach, we introduce
the ISEKAI dataset, comprising exclusively of unseen generated image-label
pairs designed for link-context learning. Extensive experiments show that our
LCL-MLLM exhibits strong link-context learning capabilities to novel concepts
over vanilla MLLMs. Code and data will be released at
https://github.com/isekai-portal/Link-Context-Learning.
LCL, a method emphasizing causal reasoning in link-context learning, enhances MLLMs' ability to recognize novel concepts and images with fewer training samples.
Multimodal Large Language Models Large Language Models In-Context Learning link-context learning causal relationship support set query set causal links ISEKAI dataset LCL-MLLM vanilla MLLMs