UniPixel: Unified Object Referring and Segmentation for Pixel-Level
  Visual Reasoning
Recent advances in Large Multi-modal Models (LMMs) have demonstrated their
remarkable success as general-purpose multi-modal assistants, with particular
focuses on holistic image- and video-language understanding. Conversely, less
attention has been given to scaling fine-grained pixel-level understanding
capabilities, where the models are expected to realize pixel-level alignment
between visual signals and language semantics. Some previous studies have
applied LMMs to related tasks such as region-level captioning and referring
expression segmentation. However, these models are limited to performing either
referring or segmentation tasks independently and fail to integrate these
fine-grained perception capabilities into visual reasoning. To bridge this gap,
we propose UniPixel, a large multi-modal model capable of flexibly
comprehending visual prompt inputs and generating mask-grounded responses. Our
model distinguishes itself by seamlessly integrating pixel-level perception
with general visual understanding capabilities. Specifically, UniPixel
processes visual prompts and generates relevant masks on demand, and performs
subsequent reasoning conditioning on these intermediate pointers during
inference, thereby enabling fine-grained pixel-level reasoning. The
effectiveness of our approach has been verified on 10 benchmarks across a
diverse set of tasks, including pixel-level referring/segmentation and
object-centric understanding in images/videos. A novel PixelQA task that
jointly requires referring, segmentation, and question answering is also
designed to verify the flexibility of our method.
68d2a61a0e215259d193b084
Ye Liu
claimed_verified
2025-09-23T14:24:36.336Z
6250eb5c1fd03f78d0ae550f
https://cdn-avatars.huggingface.co/v1/production/uploads/1649470297585-noauth.jpeg
Ye Liu
user
yeliudev
68d2a61a0e215259d193b085
Zongyang Ma
claimed_verified
2025-09-23T15:15:48.907Z
61bb00f6c4ac95d207b25f1b
/avatars/3b6eba701d64518d6f694942f5b2e9a9.svg
Zongyang Ma
user
zyma
68d2a61a0e215259d193b086
Junfu Pu
68d2a61a0e215259d193b087
Zhongang Qi
68d2a61a0e215259d193b088
Yang Wu
68d2a61a0e215259d193b089
Ying Shan
68d2a61a0e215259d193b08a
Chang Wen Chen
UniPixel: Unified Object Referring and Segmentation for Pixel-Level
  Visual Reasoning
Recent advances in Large Multi-modal Models (LMMs) have demonstrated their
remarkable success as general-purpose multi-modal assistants, with particular
focuses on holistic image- and video-language understanding. Conversely, less
attention has been given to scaling fine-grained pixel-level understanding
capabilities, where the models are expected to realize pixel-level alignment
between visual signals and language semantics. Some previous studies have
applied LMMs to related tasks such as region-level captioning and referring
expression segmentation. However, these models are limited to performing either
referring or segmentation tasks independently and fail to integrate these
fine-grained perception capabilities into visual reasoning. To bridge this gap,
we propose UniPixel, a large multi-modal model capable of flexibly
comprehending visual prompt inputs and generating mask-grounded responses. Our
model distinguishes itself by seamlessly integrating pixel-level perception
with general visual understanding capabilities. Specifically, UniPixel
processes visual prompts and generates relevant masks on demand, and performs
subsequent reasoning conditioning on these intermediate pointers during
inference, thereby enabling fine-grained pixel-level reasoning. The
effectiveness of our approach has been verified on 10 benchmarks across a
diverse set of tasks, including pixel-level referring/segmentation and
object-centric understanding in images/videos. A novel PixelQA task that
jointly requires referring, segmentation, and question answering is also
designed to verify the flexibility of our method.
UniPixel, a large multi-modal model, integrates pixel-level perception with general visual understanding, enabling fine-grained reasoning across various tasks including pixel-level referring, segmentation, and question answering.
Large Multi-modal Models LMMs pixel-level understanding pixel-level alignment visual signals language semantics region-level captioning referring expression segmentation visual reasoning mask-grounded responses pixel-level perception visual understanding capabilities pixel-level referring pixel-level segmentation object-centric understanding PixelQA task