Temporal Prompting Matters: Rethinking Referring Video Object
  Segmentation
Referring Video Object Segmentation (RVOS) aims to segment the object
referred to by the query sentence in the video. Most existing methods require
end-to-end training with dense mask annotations, which could be
computation-consuming and less scalable. In this work, we rethink the RVOS
problem and aim to investigate the key to this task. Based on existing
foundation segmentation models, we decompose the RVOS task into referring,
video, and segmentation factors, and propose a Temporal Prompt Generation and
Selection (Tenet) framework to address the referring and video factors while
leaving the segmentation problem to foundation models. To efficiently adapt
image-based foundation segmentation models to referring video object
segmentation, we leverage off-the-shelf object detectors and trackers to
produce temporal prompts associated with the referring sentence. While
high-quality temporal prompts could be produced, they can not be easily
identified from confidence scores. To tackle this issue, we propose Prompt
Preference Learning to evaluate the quality of the produced temporal prompts.
By taking such prompts to instruct image-based foundation segmentation models,
we would be able to produce high-quality masks for the referred object,
enabling efficient model adaptation to referring video object segmentation.
Experiments on RVOS benchmarks demonstrate the effectiveness of the Tenet
framework.
68e717097ae125f9582e694a
Ci-Siang Lin
68e717097ae125f9582e694b
Min-Hung Chen
claimed_verified
2025-10-09T06:50:25.264Z
64ae22dd1aee69ece065cdcd
https://cdn-avatars.huggingface.co/v1/production/uploads/64ae22dd1aee69ece065cdcd/JG7QaHIrr4i2k4uwR4pZK.png
Min-Hung Chen
user
cmhungsteve
68e717097ae125f9582e694c
I-Jieh Liu
68e717097ae125f9582e694d
Chien-Yi Wang
68e717097ae125f9582e694e
Sifei Liu
68e717097ae125f9582e694f
Yu-Chiang Frank Wang
Temporal Prompting Matters: Rethinking Referring Video Object
  Segmentation
Referring Video Object Segmentation (RVOS) aims to segment the object
referred to by the query sentence in the video. Most existing methods require
end-to-end training with dense mask annotations, which could be
computation-consuming and less scalable. In this work, we rethink the RVOS
problem and aim to investigate the key to this task. Based on existing
foundation segmentation models, we decompose the RVOS task into referring,
video, and segmentation factors, and propose a Temporal Prompt Generation and
Selection (Tenet) framework to address the referring and video factors while
leaving the segmentation problem to foundation models. To efficiently adapt
image-based foundation segmentation models to referring video object
segmentation, we leverage off-the-shelf object detectors and trackers to
produce temporal prompts associated with the referring sentence. While
high-quality temporal prompts could be produced, they can not be easily
identified from confidence scores. To tackle this issue, we propose Prompt
Preference Learning to evaluate the quality of the produced temporal prompts.
By taking such prompts to instruct image-based foundation segmentation models,
we would be able to produce high-quality masks for the referred object,
enabling efficient model adaptation to referring video object segmentation.
Experiments on RVOS benchmarks demonstrate the effectiveness of the Tenet
framework.
The Tenet framework decomposes the RVOS task into referring, video, and segmentation factors, using temporal prompts and prompt preference learning to adapt image-based foundation segmentation models for efficient RVOS.
Referring Video Object Segmentation (RVOS) Temporal Prompt Generation and Selection (Tenet) foundation segmentation models temporal prompts Prompt Preference Learning object detectors trackers