MIMIC-IT: Multi-Modal In-Context Instruction Tuning
High-quality instructions and responses are essential for the zero-shot
performance of large language models on interactive natural language tasks. For
interactive vision-language tasks involving intricate visual scenes, a large
quantity of diverse and creative instruction-response pairs should be
imperative to tune vision-language models (VLMs). Nevertheless, the current
availability of vision-language instruction-response pairs in terms of
quantity, diversity, and creativity remains limited, posing challenges to the
generalization of interactive VLMs. Here we present MultI-Modal In-Context
Instruction Tuning (MIMIC-IT), a dataset comprising 2.8 million multimodal
instruction-response pairs, with 2.2 million unique instructions derived from
images and videos. Each pair is accompanied by multi-modal in-context
information, forming conversational contexts aimed at empowering VLMs in
perception, reasoning, and planning. The instruction-response collection
process, dubbed as Syphus, is scaled using an automatic annotation pipeline
that combines human expertise with GPT's capabilities. Using the MIMIC-IT
dataset, we train a large VLM named Otter. Based on extensive evaluations
conducted on vision-language benchmarks, it has been observed that Otter
demonstrates remarkable proficiency in multi-modal perception, reasoning, and
in-context learning. Human evaluation reveals it effectively aligns with the
user's intentions. We release the MIMIC-IT dataset, instruction-response
collection pipeline, benchmarks, and the Otter model.
6482a1fa93362a0d1216f274
Bo Li
claimed_verified
2024-08-09T07:49:28.279Z
62d3f7d84b0933c48f3cdd9c
https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/rSoSyH0Td9fD9HVlyK7bh.jpeg
Bo Li
user
luodian
6482a1fa93362a0d1216f275
Yuanhan Zhang
admin_assigned
2023-06-12T13:32:49.378Z
62a993d80472c0b7f94027df
https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
Zhang Yuanhan
user
ZhangYuanhan
6482a1fa93362a0d1216f276
Liangyu Chen
claimed_verified
2023-06-12T09:12:26.078Z
62b67da0f56de4396ca9e44b
https://cdn-avatars.huggingface.co/v1/production/uploads/1658586059273-62b67da0f56de4396ca9e44b.jpeg
Liangyu Chen
user
liangyuch
6482a1fa93362a0d1216f277
Jinghao Wang
6482a1fa93362a0d1216f278
Fanyi Pu
admin_assigned
2023-06-12T13:33:45.435Z
646e1ef5075bbcc48ddf21e8
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_vJC0zeVOIvaNV2R6toqg.jpeg
Pu Fanyi
user
pufanyi
6482a1fa93362a0d1216f279
Jingkang Yang
admin_assigned
2023-06-12T13:34:01.256Z
62b5777f593a2c49da69dc02
https://cdn-avatars.huggingface.co/v1/production/uploads/1658152070753-62b5777f593a2c49da69dc02.jpeg
Jingkang Yang
user
Jingkang
6482a1fa93362a0d1216f27a
Chunyuan Li
extracted_confirmed
2023-07-11T05:13:22.469Z
62aba526cae4462c0c6caa0f
/avatars/430560ec2c2547f819225769ab432f30.svg
Chunyuan Li
user
Chunyuan24
6482a1fa93362a0d1216f27b
Ziwei Liu
admin_assigned
2023-06-12T13:34:25.660Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
MIMIC-IT: Multi-Modal In-Context Instruction Tuning
High-quality instructions and responses are essential for the zero-shot
performance of large language models on interactive natural language tasks. For
interactive vision-language tasks involving intricate visual scenes, a large
quantity of diverse and creative instruction-response pairs should be
imperative to tune vision-language models (VLMs). Nevertheless, the current
availability of vision-language instruction-response pairs in terms of
quantity, diversity, and creativity remains limited, posing challenges to the
generalization of interactive VLMs. Here we present MultI-Modal In-Context
Instruction Tuning (MIMIC-IT), a dataset comprising 2.8 million multimodal
instruction-response pairs, with 2.2 million unique instructions derived from
images and videos. Each pair is accompanied by multi-modal in-context
information, forming conversational contexts aimed at empowering VLMs in
perception, reasoning, and planning. The instruction-response collection
process, dubbed as Syphus, is scaled using an automatic annotation pipeline
that combines human expertise with GPT's capabilities. Using the MIMIC-IT
dataset, we train a large VLM named Otter. Based on extensive evaluations
conducted on vision-language benchmarks, it has been observed that Otter
demonstrates remarkable proficiency in multi-modal perception, reasoning, and
in-context learning. Human evaluation reveals it effectively aligns with the
user's intentions. We release the MIMIC-IT dataset, instruction-response
collection pipeline, benchmarks, and the Otter model.
MIMIC-IT, a multimodal instruction-response dataset, enhances VLMs' zero-shot performance on vision-language tasks through extensive and diverse data, leading to improved perception, reasoning, and planning.
MultI-Modal In-Context Instruction Tuning MIMIC-IT vision-language models VLMs multimodal instruction-response pairs conversational contexts Syphus automatic annotation pipeline GPT Otter vision-language benchmarks multi-modal perception reasoning in-context learning