Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in
  MLLMs
Multimodal large language models (MLLMs) have advanced rapidly in recent
years. However, existing approaches for vision tasks often rely on indirect
representations, such as generating coordinates as text for detection, which
limits performance and prevents dense prediction tasks like segmentation. To
overcome these challenges, we introduce Patch-as-Decodable Token (PaDT), a
unified paradigm that enables MLLMs to directly generate both textual and
diverse visual outputs. Central to PaDT are Visual Reference Tokens (VRTs),
derived from visual patch embeddings of query images and interleaved seamlessly
with LLM's output textual tokens. A lightweight decoder then transforms LLM's
outputs into detection, segmentation, and grounding predictions. Unlike prior
methods, PaDT processes VRTs independently at each forward pass and dynamically
expands the embedding table, thus improving localization and differentiation
among similar objects. We further tailor a training strategy for PaDT by
randomly selecting VRTs for supervised fine-tuning and introducing a robust
per-token cross-entropy loss. Our empirical studies across four visual
perception and understanding tasks suggest PaDT consistently achieving
state-of-the-art performance, even compared with significantly larger MLLM
models. The code is available at https://github.com/Gorilla-Lab-SCUT/PaDT.
68e64046975ac4c405ef2237
Yongyi Su
68e64046975ac4c405ef2238
Haojie Zhang
claimed_verified
2025-10-09T12:49:09.173Z
662203bcb3bb05d1d34ee728
/avatars/f7cad8e4cc04bdcc931bacf5bf802554.svg
Zhanghaojie
user
eehaojiezhang
68e64046975ac4c405ef2239
Shijie Li
68e64046975ac4c405ef223a
Nanqing Liu
68e64046975ac4c405ef223b
Jingyi Liao
68e64046975ac4c405ef223c
Junyi Pan
68e64046975ac4c405ef223d
Yuan Liu
68e64046975ac4c405ef223e
Xiaofen Xing
68e64046975ac4c405ef223f
Chong Sun
68e64046975ac4c405ef2240
Chen Li
68e64046975ac4c405ef2241
Nancy F. Chen
68e64046975ac4c405ef2242
Shuicheng Yan
68e64046975ac4c405ef2243
Xulei Yang
68e64046975ac4c405ef2244
Xun Xu
Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in
  MLLMs
Multimodal large language models (MLLMs) have advanced rapidly in recent
years. However, existing approaches for vision tasks often rely on indirect
representations, such as generating coordinates as text for detection, which
limits performance and prevents dense prediction tasks like segmentation. To
overcome these challenges, we introduce Patch-as-Decodable Token (PaDT), a
unified paradigm that enables MLLMs to directly generate both textual and
diverse visual outputs. Central to PaDT are Visual Reference Tokens (VRTs),
derived from visual patch embeddings of query images and interleaved seamlessly
with LLM's output textual tokens. A lightweight decoder then transforms LLM's
outputs into detection, segmentation, and grounding predictions. Unlike prior
methods, PaDT processes VRTs independently at each forward pass and dynamically
expands the embedding table, thus improving localization and differentiation
among similar objects. We further tailor a training strategy for PaDT by
randomly selecting VRTs for supervised fine-tuning and introducing a robust
per-token cross-entropy loss. Our empirical studies across four visual
perception and understanding tasks suggest PaDT consistently achieving
state-of-the-art performance, even compared with significantly larger MLLM
models. The code is available at https://github.com/Gorilla-Lab-SCUT/PaDT.
PaDT, a unified paradigm for multimodal large language models, directly generates both textual and visual outputs, achieving state-of-the-art performance in visual perception tasks.
Multimodal large language models Patch-as-Decodable Token PaDT Visual Reference Tokens VRTs visual patch embeddings lightweight decoder detection segmentation grounding predictions supervised fine-tuning per-token cross-entropy loss