Kosmos-2: Grounding Multimodal Large Language Models to the World
We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new
capabilities of perceiving object descriptions (e.g., bounding boxes) and
grounding text to the visual world. Specifically, we represent refer
expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where
object descriptions are sequences of location tokens. Together with multimodal
corpora, we construct large-scale data of grounded image-text pairs (called
GrIT) to train the model. In addition to the existing capabilities of MLLMs
(e.g., perceiving general modalities, following instructions, and performing
in-context learning), Kosmos-2 integrates the grounding capability into
downstream applications. We evaluate Kosmos-2 on a wide range of tasks,
including (i) multimodal grounding, such as referring expression comprehension,
and phrase grounding, (ii) multimodal referring, such as referring expression
generation, (iii) perception-language tasks, and (iv) language understanding
and generation. This work lays out the foundation for the development of
Embodiment AI and sheds light on the big convergence of language, multimodal
perception, action, and world modeling, which is a key step toward artificial
general intelligence. Data, demo, and pretrained models are available at
https://aka.ms/kosmos-2.
649a54023c50d4ec5335193f
Zhiliang Peng
claimed_verified
2023-07-04T07:24:20.810Z
646c408336505117e22f4b36
/avatars/8201da3c4ec51c8e113446f9578d44f6.svg
zhiliang
user
zzliang
649a54023c50d4ec53351940
Wenhui Wang
649a54023c50d4ec53351941
Li Dong
claimed_verified
2023-06-28T07:41:25.517Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
user
unilm
649a54023c50d4ec53351942
Yaru Hao
649a54023c50d4ec53351943
Shaohan Huang
admin_assigned
2023-06-27T13:08:34.909Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
649a54023c50d4ec53351944
Shuming Ma
admin_assigned
2023-06-27T13:08:06.367Z
613f07f40153aafa379775f2
/avatars/3965175b320d753d9a5ccb0c7d9298a4.svg
Shuming Ma
user
shumingma
649a54023c50d4ec53351945
Furu Wei
admin_assigned
2023-06-27T13:07:46.648Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
Kosmos-2: Grounding Multimodal Large Language Models to the World
We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new
capabilities of perceiving object descriptions (e.g., bounding boxes) and
grounding text to the visual world. Specifically, we represent refer
expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where
object descriptions are sequences of location tokens. Together with multimodal
corpora, we construct large-scale data of grounded image-text pairs (called
GrIT) to train the model. In addition to the existing capabilities of MLLMs
(e.g., perceiving general modalities, following instructions, and performing
in-context learning), Kosmos-2 integrates the grounding capability into
downstream applications. We evaluate Kosmos-2 on a wide range of tasks,
including (i) multimodal grounding, such as referring expression comprehension,
and phrase grounding, (ii) multimodal referring, such as referring expression
generation, (iii) perception-language tasks, and (iv) language understanding
and generation. This work lays out the foundation for the development of
Embodiment AI and sheds light on the big convergence of language, multimodal
perception, action, and world modeling, which is a key step toward artificial
general intelligence. Data, demo, and pretrained models are available at
https://aka.ms/kosmos-2.
Kosmos-2, a Multimodal Large Language Model, combines text-to-visual grounding with existing MLLM capabilities, enabling tasks like referring expression comprehension and phrase grounding.
Multimodal Large Language Model MLLM refer expressions Markdown location tokens grounded image-text pairs multimodal grounding referring expression comprehension phrase grounding multimodal referring referring expression generation perception-language tasks language understanding language generation Embodiment AI