Unlocking the Potential of MLLMs in Referring Expression Segmentation
  via a Light-weight Mask Decode
Reference Expression Segmentation (RES) aims to segment image regions
specified by referring expressions and has become popular with the rise of
multimodal large models (MLLMs). While MLLMs excel in semantic understanding,
their token-generation paradigm struggles with pixel-level dense prediction.
Existing RES methods either couple MLLMs with the parameter-heavy Segment
Anything Model (SAM) with 632M network parameters or adopt SAM-free lightweight
pipelines that sacrifice accuracy. To address the trade-off between performance
and cost, we specifically propose MLLMSeg, a novel framework that fully
exploits the inherent visual detail features encoded in the MLLM vision encoder
without introducing an extra visual encoder. Besides, we propose a
detail-enhanced and semantic-consistent feature fusion module (DSFF) that fully
integrates the detail-related visual feature with the semantic-related feature
output by the large language model (LLM) of MLLM. Finally, we establish a
light-weight mask decoder with only 34M network parameters that optimally
leverages detailed spatial features from the visual encoder and semantic
features from the LLM to achieve precise mask prediction. Extensive experiments
demonstrate that our method generally surpasses both SAM-based and SAM-free
competitors, striking a better balance between performance and cost. Code is
available at https://github.com/jcwang0602/MLLMSeg.
6894c3aa48b0ae5ca2710c22
Jingchao Wang
claimed_verified
2025-08-08T16:21:00.220Z
65eaa07cb6c760d77468b4b6
/avatars/4a1aae58986b40444351e0a167ca807c.svg
Jingchao Wang
user
jcwang0602
6894c3aa48b0ae5ca2710c23
Zhijian Wu
6894c3aa48b0ae5ca2710c24
Dingjiang Huang
6894c3aa48b0ae5ca2710c25
Yefeng Zheng
6894c3aa48b0ae5ca2710c26
Hong Wang
claimed_verified
2025-08-20T08:53:33.678Z
67205619fc1619acab55084a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1d-WhrrCUdCsXollNlyq.png
HongWang
user
hongwang01
Unlocking the Potential of MLLMs in Referring Expression Segmentation
  via a Light-weight Mask Decode
Reference Expression Segmentation (RES) aims to segment image regions
specified by referring expressions and has become popular with the rise of
multimodal large models (MLLMs). While MLLMs excel in semantic understanding,
their token-generation paradigm struggles with pixel-level dense prediction.
Existing RES methods either couple MLLMs with the parameter-heavy Segment
Anything Model (SAM) with 632M network parameters or adopt SAM-free lightweight
pipelines that sacrifice accuracy. To address the trade-off between performance
and cost, we specifically propose MLLMSeg, a novel framework that fully
exploits the inherent visual detail features encoded in the MLLM vision encoder
without introducing an extra visual encoder. Besides, we propose a
detail-enhanced and semantic-consistent feature fusion module (DSFF) that fully
integrates the detail-related visual feature with the semantic-related feature
output by the large language model (LLM) of MLLM. Finally, we establish a
light-weight mask decoder with only 34M network parameters that optimally
leverages detailed spatial features from the visual encoder and semantic
features from the LLM to achieve precise mask prediction. Extensive experiments
demonstrate that our method generally surpasses both SAM-based and SAM-free
competitors, striking a better balance between performance and cost. Code is
available at https://github.com/jcwang0602/MLLMSeg.
MLLMSeg integrates MLLM vision encoder and LLM features with a lightweight mask decoder to achieve high accuracy in reference expression segmentation with reduced computational cost.
Reference Expression Segmentation multimodal large models MLLMs Segment Anything Model SAM parameter-heavy lightweight pipelines MLLMSeg feature fusion module DSFF detail-enhanced semantic-consistent mask decoder precise mask prediction