MobileSAMv2: Faster Segment Anything to Everything
Segment anything model (SAM) addresses two practical yet challenging
segmentation tasks: segment anything (SegAny), which utilizes a
certain point to predict the mask for a single object of interest, and
segment everything (SegEvery), which predicts the masks for all
objects on the image. What makes SegAny slow for SAM is its heavyweight image
encoder, which has been addressed by MobileSAM via decoupled knowledge
distillation. The efficiency bottleneck of SegEvery with SAM, however, lies in
its mask decoder because it needs to first generate numerous masks with
redundant grid-search prompts and then perform filtering to obtain the final
valid masks. We propose to improve its efficiency by directly generating the
final masks with only valid prompts, which can be obtained through object
discovery. Our proposed approach not only helps reduce the total time on the
mask decoder by at least 16 times but also achieves superior performance.
Specifically, our approach yields an average performance boost of 3.6\% (42.5\%
v.s. 38.9\%) for zero-shot object proposal on the LVIS dataset with
the mask AR@K metric. Qualitative results show that our approach generates
fine-grained masks while avoiding over-segmenting things. This project
targeting faster SegEvery than the original SAM is termed MobileSAMv2 to
differentiate from MobileSAM which targets faster SegAny. Moreover, we
demonstrate that our new prompt sampling is also compatible with the distilled
image encoders in MobileSAM, contributing to a unified framework for efficient
SegAny and SegEvery. The code is available at the same link as MobileSAM
Project
https://github.com/ChaoningZhang/MobileSAM{red{https://github.com/ChaoningZhang/MobileSAM}}.
abstract
657fd85d9a9e347d4bfbda87
Chaoning Zhang
extracted_pending
2023-12-18T05:27:58.662Z
649a9b7cef592b26c3d39eb5
/avatars/390b7ae94de540852ad5ce46bfb148e7.svg
Zhang
user
Chaoning
657fd85d9a9e347d4bfbda88
Dongshen Han
657fd85d9a9e347d4bfbda89
Sheng Zheng
657fd85d9a9e347d4bfbda8a
Jinwoo Choi
657fd85d9a9e347d4bfbda8b
Tae-Ho Kim
admin_assigned
2023-12-18T11:06:40.592Z
64cccbe8c7f30fbf7bbf5a6d
/avatars/1c1f7c2da22c2afb3b352d446e80e5ef.svg
Tae-Ho Kim
user
ktho22
657fd85d9a9e347d4bfbda8c
Choong Seon Hong
MobileSAMv2: Faster Segment Anything to Everything
Segment anything model (SAM) addresses two practical yet challenging
segmentation tasks: segment anything (SegAny), which utilizes a
certain point to predict the mask for a single object of interest, and
segment everything (SegEvery), which predicts the masks for all
objects on the image. What makes SegAny slow for SAM is its heavyweight image
encoder, which has been addressed by MobileSAM via decoupled knowledge
distillation. The efficiency bottleneck of SegEvery with SAM, however, lies in
its mask decoder because it needs to first generate numerous masks with
redundant grid-search prompts and then perform filtering to obtain the final
valid masks. We propose to improve its efficiency by directly generating the
final masks with only valid prompts, which can be obtained through object
discovery. Our proposed approach not only helps reduce the total time on the
mask decoder by at least 16 times but also achieves superior performance.
Specifically, our approach yields an average performance boost of 3.6\% (42.5\%
v.s. 38.9\%) for zero-shot object proposal on the LVIS dataset with
the mask AR@K metric. Qualitative results show that our approach generates
fine-grained masks while avoiding over-segmenting things. This project
targeting faster SegEvery than the original SAM is termed MobileSAMv2 to
differentiate from MobileSAM which targets faster SegAny. Moreover, we
demonstrate that our new prompt sampling is also compatible with the distilled
image encoders in MobileSAM, contributing to a unified framework for efficient
SegAny and SegEvery. The code is available at the same link as MobileSAM
Project
https://github.com/ChaoningZhang/MobileSAM{red{https://github.com/ChaoningZhang/MobileSAM}}.
abstract
A new approach improves MobileSAM's efficiency for the SegEvery task by directly generating valid masks, boosting performance and reducing computation time.
segment anything segmentation tasks image encoder heavy image encoder decoupled knowledge distillation mask decoder mask generation object discovery prompt sampling LVIS dataset mask AR@$K$ metric zero-shot object proposal