Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding
Multimodal Large Language Models (MLLMs) achieve remarkable performance for
fine-grained pixel-level understanding tasks. However, all the works rely
heavily on extra components, such as vision encoder (CLIP), segmentation
experts, leading to high system complexity and limiting model scaling. In this
work, our goal is to explore a highly simplified MLLM without introducing extra
components. Our work is motivated by the recent works on Single trAnsformer as
a unified vIsion-Language Model (SAIL) design, where these works jointly learn
vision tokens and text tokens in transformers. We present Pixel-SAIL, a single
transformer for pixel-wise MLLM tasks. In particular, we present three
technical improvements on the plain baseline. First, we design a learnable
upsampling module to refine visual token features. Secondly, we propose a novel
visual prompt injection strategy to enable the single transformer to understand
visual prompt inputs and benefit from the early fusion of visual prompt
embeddings and vision tokens. Thirdly, we introduce a vision expert
distillation strategy to efficiently enhance the single transformer's
fine-grained feature extraction capability. In addition, we have collected a
comprehensive pixel understanding benchmark (PerBench), using a manual check.
It includes three tasks: detailed object description, visual prompt-based
question answering, and visual-text referring segmentation. Extensive
experiments on four referring segmentation benchmarks, one visual prompt
benchmark, and our PerBench show that our Pixel-SAIL achieves comparable or
even better results with a much simpler pipeline. Code and model will be
released at https://github.com/magic-research/Sa2VA.
67ff26c3414c03ebc1d42529
Tao Zhang
67ff26c3414c03ebc1d4252a
Xiangtai Li
claimed_verified
2025-04-16T08:42:53.598Z
63958b4414513eaf9029ebf1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
Xiangtai Li
user
LXT
67ff26c3414c03ebc1d4252b
Zilong Huang
67ff26c3414c03ebc1d4252c
Yanwei Li
67ff26c3414c03ebc1d4252d
Weixian Lei
67ff26c3414c03ebc1d4252e
Xueqing Deng
67ff26c3414c03ebc1d4252f
Shihao Chen
67ff26c3414c03ebc1d42530
Shunping Ji
67ff26c3414c03ebc1d42531
Jiashi Feng
Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding
Multimodal Large Language Models (MLLMs) achieve remarkable performance for
fine-grained pixel-level understanding tasks. However, all the works rely
heavily on extra components, such as vision encoder (CLIP), segmentation
experts, leading to high system complexity and limiting model scaling. In this
work, our goal is to explore a highly simplified MLLM without introducing extra
components. Our work is motivated by the recent works on Single trAnsformer as
a unified vIsion-Language Model (SAIL) design, where these works jointly learn
vision tokens and text tokens in transformers. We present Pixel-SAIL, a single
transformer for pixel-wise MLLM tasks. In particular, we present three
technical improvements on the plain baseline. First, we design a learnable
upsampling module to refine visual token features. Secondly, we propose a novel
visual prompt injection strategy to enable the single transformer to understand
visual prompt inputs and benefit from the early fusion of visual prompt
embeddings and vision tokens. Thirdly, we introduce a vision expert
distillation strategy to efficiently enhance the single transformer's
fine-grained feature extraction capability. In addition, we have collected a
comprehensive pixel understanding benchmark (PerBench), using a manual check.
It includes three tasks: detailed object description, visual prompt-based
question answering, and visual-text referring segmentation. Extensive
experiments on four referring segmentation benchmarks, one visual prompt
benchmark, and our PerBench show that our Pixel-SAIL achieves comparable or
even better results with a much simpler pipeline. Code and model will be
released at https://github.com/magic-research/Sa2VA.
multimodal large language models (MLLMs) pixel-level understanding vision encoder (CLIP) segmentation experts single transformer as a unified vision-language model (SAIL) pixel-wise MLLM tasks learnable upsampling module visual prompt injection visual prompt embeddings vision expert distillation pixel understanding benchmark (PerBench) detailed object description visual prompt-based question answering visual-text referring segmentation referring segmentation benchmarks visual prompt benchmark