OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and
  Understanding
Current universal segmentation methods demonstrate strong capabilities in
pixel-level image and video understanding. However, they lack reasoning
abilities and cannot be controlled via text instructions. In contrast, large
vision-language multimodal models exhibit powerful vision-based conversation
and reasoning capabilities but lack pixel-level understanding and have
difficulty accepting visual prompts for flexible user interaction. This paper
proposes OMG-LLaVA, a new and elegant framework combining powerful pixel-level
vision understanding with reasoning abilities. It can accept various visual and
text prompts for flexible user interaction. Specifically, we use a universal
segmentation method as the visual encoder, integrating image information,
perception priors, and visual prompts into visual tokens provided to the LLM.
The LLM is responsible for understanding the user's text instructions and
providing text responses and pixel-level segmentation results based on the
visual information. We propose perception prior embedding to better integrate
perception priors with image features. OMG-LLaVA achieves image-level,
object-level, and pixel-level reasoning and understanding in a single model,
matching or surpassing the performance of specialized methods on multiple
benchmarks. Rather than using LLM to connect each specialist, our work aims at
end-to-end training on one encoder, one decoder, and one LLM. The code and
model have been released for further research.
667e26bdb0f62e5e076bc54f
Tao Zhang
claimed_verified
2025-01-10T09:21:03.597Z
6436435c2d0ed796669258d3
/avatars/d357378eb039391e8ce74bbd84b80d07.svg
zhangtao
user
zhangtao-whu
667e26bdb0f62e5e076bc550
Xiangtai Li
admin_assigned
2024-06-28T09:20:50.107Z
63958b4414513eaf9029ebf1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
Xiangtai Li
user
LXT
667e26bdb0f62e5e076bc551
Hao Fei
claimed_verified
2024-11-15T09:46:32.950Z
647773a1168cb428e00e9a8f
https://cdn-avatars.huggingface.co/v1/production/uploads/647773a1168cb428e00e9a8f/NiRR3ScY6Plzjibfwy1hC.jpeg
Hao Fei
user
scofield7419
667e26bdb0f62e5e076bc552
Haobo Yuan
admin_assigned
2024-06-28T09:21:29.050Z
6391e41f2e73987364e6bcb2
/avatars/d09a9ee329bb8c3a9e2929d67d24e97d.svg
Haobo Yuan
user
HarborYuan
667e26bdb0f62e5e076bc553
Shengqiong Wu
admin_assigned
2024-06-28T09:25:21.171Z
64c139d867eff857ea51caa8
/avatars/4b7b3f41c2e2cfa21dd43bbac6e081ae.svg
Shengqiong Wu
user
ChocoWu
667e26bdb0f62e5e076bc554
Shunping Ji
667e26bdb0f62e5e076bc555
Chen Change Loy
667e26bdb0f62e5e076bc556
Shuicheng Yan
OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and
  Understanding
Current universal segmentation methods demonstrate strong capabilities in
pixel-level image and video understanding. However, they lack reasoning
abilities and cannot be controlled via text instructions. In contrast, large
vision-language multimodal models exhibit powerful vision-based conversation
and reasoning capabilities but lack pixel-level understanding and have
difficulty accepting visual prompts for flexible user interaction. This paper
proposes OMG-LLaVA, a new and elegant framework combining powerful pixel-level
vision understanding with reasoning abilities. It can accept various visual and
text prompts for flexible user interaction. Specifically, we use a universal
segmentation method as the visual encoder, integrating image information,
perception priors, and visual prompts into visual tokens provided to the LLM.
The LLM is responsible for understanding the user's text instructions and
providing text responses and pixel-level segmentation results based on the
visual information. We propose perception prior embedding to better integrate
perception priors with image features. OMG-LLaVA achieves image-level,
object-level, and pixel-level reasoning and understanding in a single model,
matching or surpassing the performance of specialized methods on multiple
benchmarks. Rather than using LLM to connect each specialist, our work aims at
end-to-end training on one encoder, one decoder, and one LLM. The code and
model have been released for further research.
OMG-LLaVA integrates pixel-level vision understanding and reasoning capabilities through a universal segmentation method combined with LLM, achieving high performance across multiple benchmarks.
universal segmentation method visual encoder visual tokens perception prior embedding image-level reasoning object-level reasoning pixel-level reasoning LLM end-to-end training