Segment Anything Meets Point Tracking
The Segment Anything Model (SAM) has established itself as a powerful
zero-shot image segmentation model, employing interactive prompts such as
points to generate masks. This paper presents SAM-PT, a method extending SAM's
capability to tracking and segmenting anything in dynamic videos. SAM-PT
leverages robust and sparse point selection and propagation techniques for mask
generation, demonstrating that a SAM-based segmentation tracker can yield
strong zero-shot performance across popular video object segmentation
benchmarks, including DAVIS, YouTube-VOS, and MOSE. Compared to traditional
object-centric mask propagation strategies, we uniquely use point propagation
to exploit local structure information that is agnostic to object semantics. We
highlight the merits of point-based tracking through direct evaluation on the
zero-shot open-world Unidentified Video Objects (UVO) benchmark. To further
enhance our approach, we utilize K-Medoids clustering for point initialization
and track both positive and negative points to clearly distinguish the target
object. We also employ multiple mask decoding passes for mask refinement and
devise a point re-initialization strategy to improve tracking accuracy. Our
code integrates different point trackers and video segmentation benchmarks and
will be released at https://github.com/SysCV/sam-pt.
64a3a60b7068337139e52c65
Frano Rajič
claimed_verified
2023-07-04T19:16:26.536Z
64a40013aab6329074397dfe
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/I13uQeHFl9ynuS_Sur6rY.png
Frano Rajič
user
m43
64a3a60b7068337139e52c66
Lei Ke
claimed_verified
2023-07-04T19:16:30.734Z
6258a6455ea3a0a9b6de3f22
/avatars/6eeed72a97fb24465e5e65583fbe50cf.svg
Lei Ke
user
lkeab
64a3a60b7068337139e52c67
Yu-Wing Tai
64a3a60b7068337139e52c68
Chi-Keung Tang
64a3a60b7068337139e52c69
Martin Danelljan
64a3a60b7068337139e52c6a
Fisher Yu
Segment Anything Meets Point Tracking
The Segment Anything Model (SAM) has established itself as a powerful
zero-shot image segmentation model, employing interactive prompts such as
points to generate masks. This paper presents SAM-PT, a method extending SAM's
capability to tracking and segmenting anything in dynamic videos. SAM-PT
leverages robust and sparse point selection and propagation techniques for mask
generation, demonstrating that a SAM-based segmentation tracker can yield
strong zero-shot performance across popular video object segmentation
benchmarks, including DAVIS, YouTube-VOS, and MOSE. Compared to traditional
object-centric mask propagation strategies, we uniquely use point propagation
to exploit local structure information that is agnostic to object semantics. We
highlight the merits of point-based tracking through direct evaluation on the
zero-shot open-world Unidentified Video Objects (UVO) benchmark. To further
enhance our approach, we utilize K-Medoids clustering for point initialization
and track both positive and negative points to clearly distinguish the target
object. We also employ multiple mask decoding passes for mask refinement and
devise a point re-initialization strategy to improve tracking accuracy. Our
code integrates different point trackers and video segmentation benchmarks and
will be released at https://github.com/SysCV/sam-pt.
SAM-PT extends SAM for video object segmentation using robust and sparse point propagation, achieving strong zero-shot performance and demonstrating advantages on benchmarks like DAVIS, YouTube-VOS, MOSE, and UVO.
zero-shot image segmentation SAM segment anything model point selection mask generation point propagation video object segmentation DAVIS YouTube-VOS MOSE Unidentified Video Objects UVO K-Medoids clustering mask refinement point re-initialization point trackers video segmentation benchmarks