DC-SAM: In-Context Segment Anything in Images and Videos via Dual
  Consistency
Given a single labeled example, in-context segmentation aims to segment
corresponding objects. This setting, known as one-shot segmentation in few-shot
learning, explores the segmentation model's generalization ability and has been
applied to various vision tasks, including scene understanding and image/video
editing. While recent Segment Anything Models have achieved state-of-the-art
results in interactive segmentation, these approaches are not directly
applicable to in-context segmentation. In this work, we propose the Dual
Consistency SAM (DC-SAM) method based on prompt-tuning to adapt SAM and SAM2
for in-context segmentation of both images and videos. Our key insights are to
enhance the features of the SAM's prompt encoder in segmentation by providing
high-quality visual prompts. When generating a mask prior, we fuse the SAM
features to better align the prompt encoder. Then, we design a cycle-consistent
cross-attention on fused features and initial visual prompts. Next, a
dual-branch design is provided by using the discriminative positive and
negative prompts in the prompt encoder. Furthermore, we design a simple
mask-tube training strategy to adopt our proposed dual consistency method into
the mask tube. Although the proposed DC-SAM is primarily designed for images,
it can be seamlessly extended to the video domain with the support of SAM2.
Given the absence of in-context segmentation in the video domain, we manually
curate and construct the first benchmark from existing video segmentation
datasets, named In-Context Video Object Segmentation (IC-VOS), to better assess
the in-context capability of the model. Extensive experiments demonstrate that
our method achieves 55.5 (+1.4) mIoU on COCO-20i, 73.0 (+1.1) mIoU on
PASCAL-5i, and a J&F score of 71.52 on the proposed IC-VOS benchmark. Our
source code and benchmark are available at https://github.com/zaplm/DC-SAM.
680afc5f2c4b584e1d786eee
Mengshi Qi
680afc5f2c4b584e1d786eef
Pengfei Zhu
claimed_verified
2025-04-25T08:34:30.195Z
66a8c8e4f5cda7b8690205ef
/avatars/5c43b7b50aeb3d1459307334ddcd1d1b.svg
Pengfei Zhu
user
zaplm
680afc5f2c4b584e1d786ef0
Xiangtai Li
admin_assigned
2025-04-29T12:42:09.686Z
63958b4414513eaf9029ebf1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
Xiangtai Li
user
LXT
680afc5f2c4b584e1d786ef1
Xiaoyang Bi
admin_assigned
2025-04-29T12:42:15.469Z
659054f889145cbc7cc97978
/avatars/74b7db0e170123e0df5247d64b2fbaab.svg
Xiaoyang Bi
user
xybi
680afc5f2c4b584e1d786ef2
Lu Qi
680afc5f2c4b584e1d786ef3
Huadong Ma
680afc5f2c4b584e1d786ef4
Ming-Hsuan Yang
DC-SAM: In-Context Segment Anything in Images and Videos via Dual
  Consistency
Given a single labeled example, in-context segmentation aims to segment
corresponding objects. This setting, known as one-shot segmentation in few-shot
learning, explores the segmentation model's generalization ability and has been
applied to various vision tasks, including scene understanding and image/video
editing. While recent Segment Anything Models have achieved state-of-the-art
results in interactive segmentation, these approaches are not directly
applicable to in-context segmentation. In this work, we propose the Dual
Consistency SAM (DC-SAM) method based on prompt-tuning to adapt SAM and SAM2
for in-context segmentation of both images and videos. Our key insights are to
enhance the features of the SAM's prompt encoder in segmentation by providing
high-quality visual prompts. When generating a mask prior, we fuse the SAM
features to better align the prompt encoder. Then, we design a cycle-consistent
cross-attention on fused features and initial visual prompts. Next, a
dual-branch design is provided by using the discriminative positive and
negative prompts in the prompt encoder. Furthermore, we design a simple
mask-tube training strategy to adopt our proposed dual consistency method into
the mask tube. Although the proposed DC-SAM is primarily designed for images,
it can be seamlessly extended to the video domain with the support of SAM2.
Given the absence of in-context segmentation in the video domain, we manually
curate and construct the first benchmark from existing video segmentation
datasets, named In-Context Video Object Segmentation (IC-VOS), to better assess
the in-context capability of the model. Extensive experiments demonstrate that
our method achieves 55.5 (+1.4) mIoU on COCO-20i, 73.0 (+1.1) mIoU on
PASCAL-5i, and a J&F score of 71.52 on the proposed IC-VOS benchmark. Our
source code and benchmark are available at https://github.com/zaplm/DC-SAM.
The Dual Consistency SAM (DC-SAM) method enhances prompt-based feature alignment and introduces dual-branch and cycle-consistent cross-attention to improve in-context segmentation for both images and videos, achieving state-of-the-art results.
prompt-tuning SAM SAM2 in-context segmentation mask prior prompt encoder visual prompts cycle-consistent cross-attention dual-branch design mask-tube training In-Context Video Object Segmentation (IC-VOS) mIoU J&F score