REF-VLM: Triplet-Based Referring Paradigm for Unified Visual Decoding
Multimodal Large Language Models (MLLMs) demonstrate robust zero-shot
capabilities across diverse vision-language tasks after training on mega-scale
datasets. However, dense prediction tasks, such as semantic segmentation and
keypoint detection, pose significant challenges for MLLMs when represented
solely as text outputs. Simultaneously, current MLLMs utilizing latent
embeddings for visual task decoding generally demonstrate limited adaptability
to both multi-task learning and multi-granularity scenarios. In this work, we
present REF-VLM, an end-to-end framework for unified training of various visual
decoding tasks. To address complex visual decoding scenarios, we introduce the
Triplet-Based Referring Paradigm (TRP), which explicitly decouples three
critical dimensions in visual decoding tasks through a triplet structure:
concepts, decoding types, and targets. TRP employs symbolic delimiters to
enforce structured representation learning, enhancing the parsability and
interpretability of model outputs. Additionally, we construct Visual-Task
Instruction Following Dataset (VTInstruct), a large-scale multi-task dataset
containing over 100 million multimodal dialogue samples across 25 task types.
Beyond text inputs and outputs, VT-Instruct incorporates various visual prompts
such as point, box, scribble, and mask, and generates outputs composed of text
and visual units like box, keypoint, depth and mask. The combination of
different visual prompts and visual units generates a wide variety of task
types, expanding the applicability of REF-VLM significantly. Both qualitative
and quantitative experiments demonstrate that our REF-VLM outperforms other
MLLMs across a variety of standard benchmarks. The code, dataset, and demo
available at https://github.com/MacavityT/REF-VLM.
67d050c28db3ce9654a56a96
Yan Tai
claimed_verified
2025-03-21T11:46:36.514Z
647857b6c43296134a434fc6
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Cs9p9xSXFlP3Y24JNV7Am.jpeg
taiyan
user
Amoik
67d050c28db3ce9654a56a97
Luhao Zhu
67d050c28db3ce9654a56a98
Zhiqiang Chen
67d050c28db3ce9654a56a99
Ynan Ding
67d050c28db3ce9654a56a9a
Yiying Dong
67d050c28db3ce9654a56a9b
Xiaohong Liu
67d050c28db3ce9654a56a9c
Guodong Guo
REF-VLM: Triplet-Based Referring Paradigm for Unified Visual Decoding
Multimodal Large Language Models (MLLMs) demonstrate robust zero-shot
capabilities across diverse vision-language tasks after training on mega-scale
datasets. However, dense prediction tasks, such as semantic segmentation and
keypoint detection, pose significant challenges for MLLMs when represented
solely as text outputs. Simultaneously, current MLLMs utilizing latent
embeddings for visual task decoding generally demonstrate limited adaptability
to both multi-task learning and multi-granularity scenarios. In this work, we
present REF-VLM, an end-to-end framework for unified training of various visual
decoding tasks. To address complex visual decoding scenarios, we introduce the
Triplet-Based Referring Paradigm (TRP), which explicitly decouples three
critical dimensions in visual decoding tasks through a triplet structure:
concepts, decoding types, and targets. TRP employs symbolic delimiters to
enforce structured representation learning, enhancing the parsability and
interpretability of model outputs. Additionally, we construct Visual-Task
Instruction Following Dataset (VTInstruct), a large-scale multi-task dataset
containing over 100 million multimodal dialogue samples across 25 task types.
Beyond text inputs and outputs, VT-Instruct incorporates various visual prompts
such as point, box, scribble, and mask, and generates outputs composed of text
and visual units like box, keypoint, depth and mask. The combination of
different visual prompts and visual units generates a wide variety of task
types, expanding the applicability of REF-VLM significantly. Both qualitative
and quantitative experiments demonstrate that our REF-VLM outperforms other
MLLMs across a variety of standard benchmarks. The code, dataset, and demo
available at https://github.com/MacavityT/REF-VLM.
REF-VLM, an end-to-end framework using the Triplet-Based Referring Paradigm, enhances performance in diverse visual decoding tasks by decoupling concepts, decoding types, and targets in multimodal large language models.
Multimodal Large Language Models MLMMs semantic segmentation keypoint detection latent embeddings REF-VLM Triplet-Based Referring Paradigm TRP structured representation learning Visual-Task Instruction Following Dataset VTInstruct multimodal dialogue visual prompts visual units