LEGO:Language Enhanced Multi-modal Grounding Model
Multi-modal large language models have demonstrated impressive performance
across various tasks in different modalities. However, existing multi-modal
models primarily emphasize capturing global information within each modality
while neglecting the importance of perceiving local information across
modalities. Consequently, these models lack the ability to effectively
understand the fine-grained details of input data, limiting their performance
in tasks that require a more nuanced understanding. To address this limitation,
there is a compelling need to develop models that enable fine-grained
understanding across multiple modalities, thereby enhancing their applicability
to a wide range of tasks. In this paper, we propose LEGO, a language enhanced
multi-modal grounding model. Beyond capturing global information like other
multi-modal models, our proposed model excels at tasks demanding a detailed
understanding of local information within the input. It demonstrates precise
identification and localization of specific regions in images or moments in
videos. To achieve this objective, we design a diversified dataset construction
pipeline, resulting in a multi-modal, multi-granularity dataset for model
training. The code, dataset, and demo of our model can be found at https:
//github.com/lzw-lzw/LEGO.
65a0a7315dfd8b9b1f288fe6
Zhaowei Li
claimed_verified
2024-09-30T08:10:45.647Z
6469eff38eaf6dcec22e0a8e
/avatars/0e73f4fba981fa0a93b6ae44ace9dbd7.svg
lizhaowei
user
zwli
65a0a7315dfd8b9b1f288fe7
Qi Xu
65a0a7315dfd8b9b1f288fe8
Dong Zhang
claimed_verified
2024-03-02T19:14:18.120Z
6377596f82ce7c3b8a528f72
/avatars/67a5ce6cd105ad90736881d93b457818.svg
Dong Zhang
user
nutation
65a0a7315dfd8b9b1f288fe9
Hang Song
65a0a7315dfd8b9b1f288fea
Yiqing Cai
admin_assigned
2024-01-12T10:30:47.851Z
63034e3e0ddba5a5b9f47814
/avatars/c740335400421904c06d1c576cb8ae87.svg
Yiqing Cai
user
SmallCai
65a0a7315dfd8b9b1f288feb
Qi Qi
65a0a7315dfd8b9b1f288fec
Ran Zhou
65a0a7315dfd8b9b1f288fed
Junting Pan
admin_assigned
2024-01-12T10:30:22.022Z
639c2d36e62c431aa8d834a7
/avatars/a23d57dcb190711494a2a7e75e4cb9a8.svg
Junting Pan
user
juntingpan
65a0a7315dfd8b9b1f288fee
Zefeng Li
65a0a7315dfd8b9b1f288fef
Van Tu Vu
65a0a7315dfd8b9b1f288ff0
Zhida Huang
65a0a7315dfd8b9b1f288ff1
Tao Wang
LEGO:Language Enhanced Multi-modal Grounding Model
Multi-modal large language models have demonstrated impressive performance
across various tasks in different modalities. However, existing multi-modal
models primarily emphasize capturing global information within each modality
while neglecting the importance of perceiving local information across
modalities. Consequently, these models lack the ability to effectively
understand the fine-grained details of input data, limiting their performance
in tasks that require a more nuanced understanding. To address this limitation,
there is a compelling need to develop models that enable fine-grained
understanding across multiple modalities, thereby enhancing their applicability
to a wide range of tasks. In this paper, we propose LEGO, a language enhanced
multi-modal grounding model. Beyond capturing global information like other
multi-modal models, our proposed model excels at tasks demanding a detailed
understanding of local information within the input. It demonstrates precise
identification and localization of specific regions in images or moments in
videos. To achieve this objective, we design a diversified dataset construction
pipeline, resulting in a multi-modal, multi-granularity dataset for model
training. The code, dataset, and demo of our model can be found at https:
//github.com/lzw-lzw/LEGO.
LEGO is a multi-modal grounding model that improves fine-grained understanding and localization of specific regions in images or moments in videos.
multi-modal large language models LEGO multi-modal grounding model fine-grained understanding local information dataset construction pipeline multi-modal multi-granularity dataset