Deforming Videos to Masks: Flow Matching for Referring Video
  Segmentation
Referring Video Object Segmentation (RVOS) requires segmenting specific
objects in a video guided by a natural language description. The core challenge
of RVOS is to anchor abstract linguistic concepts onto a specific set of pixels
and continuously segment them through the complex dynamics of a video. Faced
with this difficulty, prior work has often decomposed the task into a pragmatic
`locate-then-segment' pipeline. However, this cascaded design creates an
information bottleneck by simplifying semantics into coarse geometric prompts
(e.g, point), and struggles to maintain temporal consistency as the segmenting
process is often decoupled from the initial language grounding. To overcome
these fundamental limitations, we propose FlowRVS, a novel framework that
reconceptualizes RVOS as a conditional continuous flow problem. This allows us
to harness the inherent strengths of pretrained T2V models, fine-grained pixel
control, text-video semantic alignment, and temporal coherence. Instead of
conventional generating from noise to mask or directly predicting mask, we
reformulate the task by learning a direct, language-guided deformation from a
video's holistic representation to its target mask. Our one-stage, generative
approach achieves new state-of-the-art results across all major RVOS
benchmarks. Specifically, achieving a J&F of 51.1 in
MeViS (+1.6 over prior SOTA) and 73.3 in the zero shot Ref-DAVIS17 (+2.7),
demonstrating the significant potential of modeling video understanding tasks
as continuous deformation processes.
68e5c2b6975ac4c405ef20a8
Zanyi Wang
68e5c2b6975ac4c405ef20a9
Dengyang Jiang
68e5c2b6975ac4c405ef20aa
Liuzhuozheng Li
claimed_verified
2025-10-10T09:38:09.388Z
670fb4273231cb3ef3b970d2
/avatars/14bb9a7bd887e358a3a447ea2847e07a.svg
liuzhuozheng Li
user
Jyugatsu
68e5c2b6975ac4c405ef20ab
Sizhe Dang
68e5c2b6975ac4c405ef20ac
Chengzu Li
68e5c2b6975ac4c405ef20ad
Harry Yang
68e5c2b6975ac4c405ef20ae
Guang Dai
68e5c2b6975ac4c405ef20af
Mengmeng Wang
68e5c2b6975ac4c405ef20b0
Jingdong Wang
Deforming Videos to Masks: Flow Matching for Referring Video
  Segmentation
Referring Video Object Segmentation (RVOS) requires segmenting specific
objects in a video guided by a natural language description. The core challenge
of RVOS is to anchor abstract linguistic concepts onto a specific set of pixels
and continuously segment them through the complex dynamics of a video. Faced
with this difficulty, prior work has often decomposed the task into a pragmatic
`locate-then-segment' pipeline. However, this cascaded design creates an
information bottleneck by simplifying semantics into coarse geometric prompts
(e.g, point), and struggles to maintain temporal consistency as the segmenting
process is often decoupled from the initial language grounding. To overcome
these fundamental limitations, we propose FlowRVS, a novel framework that
reconceptualizes RVOS as a conditional continuous flow problem. This allows us
to harness the inherent strengths of pretrained T2V models, fine-grained pixel
control, text-video semantic alignment, and temporal coherence. Instead of
conventional generating from noise to mask or directly predicting mask, we
reformulate the task by learning a direct, language-guided deformation from a
video's holistic representation to its target mask. Our one-stage, generative
approach achieves new state-of-the-art results across all major RVOS
benchmarks. Specifically, achieving a J&F of 51.1 in
MeViS (+1.6 over prior SOTA) and 73.3 in the zero shot Ref-DAVIS17 (+2.7),
demonstrating the significant potential of modeling video understanding tasks
as continuous deformation processes.
FlowRVS addresses the challenges of Referring Video Object Segmentation by reformulating the task as a continuous flow problem, leveraging pretrained T2V models and achieving state-of-the-art results.
Referring Video Object Segmentation RVOS natural language description pixel control text-video semantic alignment temporal coherence FlowRVS T2V models continuous flow problem generative approach $\mathcal{J}\&\mathcal{F}$ MeViS Ref-DAVIS17