Tracking Anything with Decoupled Video Segmentation
Training data for video segmentation are expensive to annotate. This impedes
extensions of end-to-end algorithms to new video segmentation tasks, especially
in large-vocabulary settings. To 'track anything' without training on video
data for every individual task, we develop a decoupled video segmentation
approach (DEVA), composed of task-specific image-level segmentation and
class/task-agnostic bi-directional temporal propagation. Due to this design, we
only need an image-level model for the target task (which is cheaper to train)
and a universal temporal propagation model which is trained once and
generalizes across tasks. To effectively combine these two modules, we use
bi-directional propagation for (semi-)online fusion of segmentation hypotheses
from different frames to generate a coherent segmentation. We show that this
decoupled formulation compares favorably to end-to-end approaches in several
data-scarce tasks including large-vocabulary video panoptic segmentation,
open-world video segmentation, referring video segmentation, and unsupervised
video object segmentation. Code is available at:
https://hkchengrex.github.io/Tracking-Anything-with-DEVA
64faba2c8c21ebb3db635594
Ho Kei Cheng
admin_assigned
2023-09-08T11:39:09.024Z
63041b541dd5d3c62486c294
/avatars/a5286d562f7b9082730f760e66c3bf29.svg
Ho Kei Cheng
user
hkchengrex
64faba2c8c21ebb3db635595
Seoung Wug Oh
64faba2c8c21ebb3db635596
Brian Price
64faba2c8c21ebb3db635597
Alexander Schwing
64faba2c8c21ebb3db635598
Joon-Young Lee
admin_assigned
2023-09-08T11:39:49.340Z
64b76c7990b38df83381824b
/avatars/0a5bec2ea480fb3f43c4b24d59c50e81.svg
Joon-Young Lee
user
joonyounglee
Tracking Anything with Decoupled Video Segmentation
Training data for video segmentation are expensive to annotate. This impedes
extensions of end-to-end algorithms to new video segmentation tasks, especially
in large-vocabulary settings. To 'track anything' without training on video
data for every individual task, we develop a decoupled video segmentation
approach (DEVA), composed of task-specific image-level segmentation and
class/task-agnostic bi-directional temporal propagation. Due to this design, we
only need an image-level model for the target task (which is cheaper to train)
and a universal temporal propagation model which is trained once and
generalizes across tasks. To effectively combine these two modules, we use
bi-directional propagation for (semi-)online fusion of segmentation hypotheses
from different frames to generate a coherent segmentation. We show that this
decoupled formulation compares favorably to end-to-end approaches in several
data-scarce tasks including large-vocabulary video panoptic segmentation,
open-world video segmentation, referring video segmentation, and unsupervised
video object segmentation. Code is available at:
https://hkchengrex.github.io/Tracking-Anything-with-DEVA
DEVA, a decoupled approach using image-level segmentation and bi-directional temporal propagation, achieves favorable results in various data-scarce video segmentation tasks with reduced annotation and training costs.
video segmentation task-specific image-level segmentation bi-directional temporal propagation large-vocabulary settings panoptic segmentation open-world segmentation referring video segmentation unsupervised video object segmentation