Segment and Caption Anything
We propose a method to efficiently equip the Segment Anything Model (SAM)
with the ability to generate regional captions. SAM presents strong
generalizability to segment anything while is short for semantic understanding.
By introducing a lightweight query-based feature mixer, we align the
region-specific features with the embedding space of language models for later
caption generation. As the number of trainable parameters is small (typically
in the order of tens of millions), it costs less computation, less memory
usage, and less communication bandwidth, resulting in both fast and scalable
training. To address the scarcity problem of regional caption data, we propose
to first pre-train our model on objection detection and segmentation tasks. We
call this step weak supervision pretraining since the pre-training data only
contains category names instead of full-sentence descriptions. The weak
supervision pretraining allows us to leverage many publicly available object
detection and segmentation datasets. We conduct extensive experiments to
demonstrate the superiority of our method and validate each design choice. This
work serves as a stepping stone towards scaling up regional captioning data and
sheds light on exploring efficient ways to augment SAM with regional semantics.
The project page, along with the associated code, can be accessed via the
following https://xk-huang.github.io/segment-caption-anything/.
656e960cc94d150e5de4994a
Xiaoke Huang
admin_assigned
2023-12-05T08:56:36.086Z
63318b2349a9563915469f3b
https://cdn-avatars.huggingface.co/v1/production/uploads/63318b2349a9563915469f3b/zlbeB2997i8YkoyOTb9FL.jpeg
Xiaoke Huang
user
xk-huang
656e960cc94d150e5de4994b
Jianfeng Wang
admin_assigned
2023-12-05T08:56:59.745Z
641404e74e5305c14f269b3c
/avatars/0dbe862cde7e4bef25bedf860584d8c1.svg
Jianfeng Wang
user
amsword8
656e960cc94d150e5de4994c
Yansong Tang
656e960cc94d150e5de4994d
Zheng Zhang
656e960cc94d150e5de4994e
Han Hu
656e960cc94d150e5de4994f
Jiwen Lu
656e960cc94d150e5de49950
Lijuan Wang
admin_assigned
2023-12-05T08:58:44.759Z
6413521d4e5305c14f22e110
/avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
Lijuan Wang
user
Lijuan
656e960cc94d150e5de49951
Zicheng Liu
admin_assigned
2023-12-05T08:58:53.583Z
6569f6cb44ce94a70187f407
/avatars/414ecc9739ed23f9f395bd7cfa65055f.svg
Zicheng Liu
user
ZCLiu35
Segment and Caption Anything
We propose a method to efficiently equip the Segment Anything Model (SAM)
with the ability to generate regional captions. SAM presents strong
generalizability to segment anything while is short for semantic understanding.
By introducing a lightweight query-based feature mixer, we align the
region-specific features with the embedding space of language models for later
caption generation. As the number of trainable parameters is small (typically
in the order of tens of millions), it costs less computation, less memory
usage, and less communication bandwidth, resulting in both fast and scalable
training. To address the scarcity problem of regional caption data, we propose
to first pre-train our model on objection detection and segmentation tasks. We
call this step weak supervision pretraining since the pre-training data only
contains category names instead of full-sentence descriptions. The weak
supervision pretraining allows us to leverage many publicly available object
detection and segmentation datasets. We conduct extensive experiments to
demonstrate the superiority of our method and validate each design choice. This
work serves as a stepping stone towards scaling up regional captioning data and
sheds light on exploring efficient ways to augment SAM with regional semantics.
The project page, along with the associated code, can be accessed via the
following https://xk-huang.github.io/segment-caption-anything/.
A method enhances the Segment Anything Model (SAM) with regional caption generation using a lightweight query-based feature mixer, enabling efficient use of weak supervision pretraining on publicly available datasets.
Segment Anything Model SAM regional captions query-based feature mixer embedding space language models weak supervision pretraining object detection segmentation regional semantics