Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual
  Prompt Instruction Tuning
Large Multimodal Models (LMMs) have made significant breakthroughs with the
advancement of instruction tuning. However, while existing models can
understand images and videos at a holistic level, they still struggle with
instance-level understanding that requires a more nuanced comprehension and
alignment. Instance-level understanding is crucial, as it focuses on the
specific elements that we are most interested in. Excitingly, existing works
find that the state-of-the-art LMMs exhibit strong instance understanding
capabilities when provided with explicit visual cues. Motivated by this, we
introduce an automated annotation pipeline assisted by GPT-4o to extract
instance-level information from images and videos through explicit visual
prompting for instance guidance. Building upon this pipeline, we proposed
Inst-IT, a solution to enhance LMMs in Instance understanding via explicit
visual prompt Instruction Tuning. Inst-IT consists of a benchmark to diagnose
multimodal instance-level understanding, a large-scale instruction-tuning
dataset, and a continuous instruction-tuning training paradigm to effectively
enhance spatial-temporal instance understanding capabilities of existing LMMs.
Experimental results show that, with the boost of Inst-IT, our models not only
achieve outstanding performance on Inst-IT Bench but also demonstrate
significant improvements across various generic image and video understanding
benchmarks. This highlights that our dataset not only boosts instance-level
understanding but also strengthens the overall capabilities of generic image
and video comprehension.
67519ac329945767b6c2ceeb
Wujian Peng
claimed_verified
2024-12-05T12:22:26.413Z
649bce4f200e2dff194d9883
/avatars/b55a8bdc6f7e2bf9de5f26dc1d87bee3.svg
Wujian Peng
user
wjpoom
67519ac329945767b6c2ceec
Lingchen Meng
claimed_verified
2024-12-16T09:42:49.150Z
64d852a4bab152b2470bf96e
/avatars/074ef416ea865ca97c208ab2ba55d622.svg
Lingchen Meng
user
menglc
67519ac329945767b6c2ceed
Yitong Chen
claimed_verified
2025-03-25T08:22:20.812Z
64651db3611ae99d14d392ea
/avatars/b818dc0dddc999758ab5737d5053e8c3.svg
cyt
user
Row11n
67519ac329945767b6c2ceee
Yiweng Xie
67519ac329945767b6c2ceef
Yang Liu
67519ac329945767b6c2cef0
Tao Gui
67519ac329945767b6c2cef1
Hang Xu
67519ac329945767b6c2cef2
Xipeng Qiu
67519ac329945767b6c2cef3
Zuxuan Wu
67519ac329945767b6c2cef4
Yu-Gang Jiang
Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual
  Prompt Instruction Tuning
Large Multimodal Models (LMMs) have made significant breakthroughs with the
advancement of instruction tuning. However, while existing models can
understand images and videos at a holistic level, they still struggle with
instance-level understanding that requires a more nuanced comprehension and
alignment. Instance-level understanding is crucial, as it focuses on the
specific elements that we are most interested in. Excitingly, existing works
find that the state-of-the-art LMMs exhibit strong instance understanding
capabilities when provided with explicit visual cues. Motivated by this, we
introduce an automated annotation pipeline assisted by GPT-4o to extract
instance-level information from images and videos through explicit visual
prompting for instance guidance. Building upon this pipeline, we proposed
Inst-IT, a solution to enhance LMMs in Instance understanding via explicit
visual prompt Instruction Tuning. Inst-IT consists of a benchmark to diagnose
multimodal instance-level understanding, a large-scale instruction-tuning
dataset, and a continuous instruction-tuning training paradigm to effectively
enhance spatial-temporal instance understanding capabilities of existing LMMs.
Experimental results show that, with the boost of Inst-IT, our models not only
achieve outstanding performance on Inst-IT Bench but also demonstrate
significant improvements across various generic image and video understanding
benchmarks. This highlights that our dataset not only boosts instance-level
understanding but also strengthens the overall capabilities of generic image
and video comprehension.
Inst-IT enhances large multimodal models' instance-level understanding through an automated annotation pipeline with explicit visual prompts, improving performance on specific elements in images and videos.
GPT-4o instance-level understanding automated annotation pipeline explicit visual prompting Inst-IT benchmark large-scale instruction-tuning dataset continuous instruction-tuning training paradigm spatial-temporal instance understanding