Faster Segment Anything: Towards Lightweight SAM for Mobile Applications
Segment anything model (SAM) is a prompt-guided vision foundation model for
cutting out the object of interest from its background. Since Meta research
team released the SA project, SAM has attracted significant attention due to
its impressive zero-shot transfer performance and high versatility of being
compatible with other models for advanced vision applications like image
editing with fine-grained control. Many of such use cases need to be run on
resource-constraint edge devices, like mobile Apps. In this work, we aim to
make SAM mobile-friendly by replacing the heavyweight image encoder with a
lightweight one. A naive way to train such a new SAM as in the original SAM
paper leads to unsatisfactory performance, especially when limited training
sources are available. We find that this is mainly caused by the coupled
optimization of the image encoder and mask decoder, motivated by which we
propose decoupled distillation. Concretely, we distill the knowledge from the
image encoder ViT-H in the original SAM to a lightweight image encoder, which
can be automatically compatible with the mask decoder in the original SAM. The
training can be completed on a single GPU within less than one day, and the
resulting lightweight SAM is termed MobileSAM which is more than 60 times
smaller yet performs on par with the original SAM. For inference speed,
MobileSAM runs around 10ms per image: 8ms on the image encoder and 2ms on the
mask decoder. With superior performance and a higher versatility, our MobileSAM
is 7 times smaller and 4 times faster than the concurrent FastSAM, making it
more suitable for mobile applications. The code for MobileSAM project is
provided at https://github.com/ChaoningZhang/MobileSAM
649a870096d5747b35e7535d
Chaoning Zhang
claimed_verified
2023-06-28T09:25:28.697Z
649a9b7cef592b26c3d39eb5
/avatars/390b7ae94de540852ad5ce46bfb148e7.svg
Zhang
user
Chaoning
649a870096d5747b35e7535e
Dongshen Han
649a870096d5747b35e7535f
Yu Qiao
claimed_verified
2023-06-28T20:15:24.018Z
649bfed69ff0096550981c90
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/DFABS5u1v2DQ0vZ82V501.jpeg
Yu Qiao
user
qiaoyu1002
649a870096d5747b35e75360
Jung Uk Kim
649a870096d5747b35e75361
Sung-Ho Bae
claimed_verified
2023-06-28T20:15:21.809Z
649bf062f8183583a4f1dec6
/avatars/4317ffc9c2af6739cd27fd80aa1285b3.svg
Sung Ho Bae
user
shbae84
649a870096d5747b35e75362
Seungkyu Lee
649a870096d5747b35e75363
Choong Seon Hong
Faster Segment Anything: Towards Lightweight SAM for Mobile Applications
Segment anything model (SAM) is a prompt-guided vision foundation model for
cutting out the object of interest from its background. Since Meta research
team released the SA project, SAM has attracted significant attention due to
its impressive zero-shot transfer performance and high versatility of being
compatible with other models for advanced vision applications like image
editing with fine-grained control. Many of such use cases need to be run on
resource-constraint edge devices, like mobile Apps. In this work, we aim to
make SAM mobile-friendly by replacing the heavyweight image encoder with a
lightweight one. A naive way to train such a new SAM as in the original SAM
paper leads to unsatisfactory performance, especially when limited training
sources are available. We find that this is mainly caused by the coupled
optimization of the image encoder and mask decoder, motivated by which we
propose decoupled distillation. Concretely, we distill the knowledge from the
image encoder ViT-H in the original SAM to a lightweight image encoder, which
can be automatically compatible with the mask decoder in the original SAM. The
training can be completed on a single GPU within less than one day, and the
resulting lightweight SAM is termed MobileSAM which is more than 60 times
smaller yet performs on par with the original SAM. For inference speed,
MobileSAM runs around 10ms per image: 8ms on the image encoder and 2ms on the
mask decoder. With superior performance and a higher versatility, our MobileSAM
is 7 times smaller and 4 times faster than the concurrent FastSAM, making it
more suitable for mobile applications. The code for MobileSAM project is
provided at https://github.com/ChaoningZhang/MobileSAM
MobileSAM enhances the Segment Anything Model (SAM) for mobile devices by decoupling the image encoder and mask decoder to create a lightweight version that maintains performance while significantly reducing size and improving speed.
prompt-guided vision foundation model zero-shot transfer image editing edge devices image encoder mask decoder decoupled distillation ViT-H MobileSAM inference speed FastSAM