ARGenSeg: Image Segmentation with Autoregressive Image Generation Model
We propose a novel AutoRegressive Generation-based paradigm for image
Segmentation (ARGenSeg), achieving multimodal understanding and pixel-level
perception within a unified framework. Prior works integrating image
segmentation into multimodal large language models (MLLMs) typically employ
either boundary points representation or dedicated segmentation heads. These
methods rely on discrete representations or semantic prompts fed into
task-specific decoders, which limits the ability of the MLLM to capture
fine-grained visual details. To address these challenges, we introduce a
segmentation framework for MLLM based on image generation, which naturally
produces dense masks for target objects. We leverage MLLM to output visual
tokens and detokenize them into images using an universal VQ-VAE, making the
segmentation fully dependent on the pixel-level understanding of the MLLM. To
reduce inference latency, we employ a next-scale-prediction strategy to
generate required visual tokens in parallel. Extensive experiments demonstrate
that our method surpasses prior state-of-the-art approaches on multiple
segmentation datasets with a remarkable boost in inference speed, while
maintaining strong understanding capabilities.
68fae536f158a71c5a2f5848
Xiaolong Wang
claimed_verified
2025-10-24T16:11:26.153Z
64e848dd9a3cd93b371166cf
/avatars/6fcee1fe59624113fcf233caf0197729.svg
Xiaolong Wang
user
Xiaolong-Wang
68fae536f158a71c5a2f5849
Lixiang Ru
68fae536f158a71c5a2f584a
Ziyuan Huang
68fae536f158a71c5a2f584b
Kaixiang Ji
68fae536f158a71c5a2f584c
Dandan Zheng
68fae536f158a71c5a2f584d
Jingdong Chen
68fae536f158a71c5a2f584e
Jun Zhou
ARGenSeg: Image Segmentation with Autoregressive Image Generation Model
We propose a novel AutoRegressive Generation-based paradigm for image
Segmentation (ARGenSeg), achieving multimodal understanding and pixel-level
perception within a unified framework. Prior works integrating image
segmentation into multimodal large language models (MLLMs) typically employ
either boundary points representation or dedicated segmentation heads. These
methods rely on discrete representations or semantic prompts fed into
task-specific decoders, which limits the ability of the MLLM to capture
fine-grained visual details. To address these challenges, we introduce a
segmentation framework for MLLM based on image generation, which naturally
produces dense masks for target objects. We leverage MLLM to output visual
tokens and detokenize them into images using an universal VQ-VAE, making the
segmentation fully dependent on the pixel-level understanding of the MLLM. To
reduce inference latency, we employ a next-scale-prediction strategy to
generate required visual tokens in parallel. Extensive experiments demonstrate
that our method surpasses prior state-of-the-art approaches on multiple
segmentation datasets with a remarkable boost in inference speed, while
maintaining strong understanding capabilities.
A novel AutoRegressive Generation-based paradigm for image segmentation leverages multimodal large language models and VQ-VAE for dense mask generation, achieving high performance and fast inference.
AutoRegressive Generation image segmentation multimodal large language models boundary points representation segmentation heads discrete representations semantic prompts task-specific decoders visual tokens universal VQ-VAE next-scale-prediction strategy