FrozenSeg: Harmonizing Frozen Foundation Models for Open-Vocabulary
  Segmentation
Open-vocabulary segmentation poses significant challenges, as it requires
segmenting and recognizing objects across an open set of categories in
unconstrained environments. Building on the success of powerful vision-language
(ViL) foundation models, such as CLIP, recent efforts sought to harness their
zero-short capabilities to recognize unseen categories. Despite notable
performance improvements, these models still encounter the critical issue of
generating precise mask proposals for unseen categories and scenarios,
resulting in inferior segmentation performance eventually. To address this
challenge, we introduce a novel approach, FrozenSeg, designed to integrate
spatial knowledge from a localization foundation model (e.g., SAM) and semantic
knowledge extracted from a ViL model (e.g., CLIP), in a synergistic framework.
Taking the ViL model's visual encoder as the feature backbone, we inject the
space-aware feature into the learnable queries and CLIP features within the
transformer decoder. In addition, we devise a mask proposal ensemble strategy
for further improving the recall rate and mask quality. To fully exploit
pre-trained knowledge while minimizing training overhead, we freeze both
foundation models, focusing optimization efforts solely on a lightweight
transformer decoder for mask proposal generation-the performance bottleneck.
Extensive experiments demonstrate that FrozenSeg advances state-of-the-art
results across various segmentation benchmarks, trained exclusively on COCO
panoptic data, and tested in a zero-shot manner. Code is available at
https://github.com/chenxi52/FrozenSeg.
66dad96e75b3337a858eed29
Xi Chen
66dad96e75b3337a858eed2a
Haosen Yang
claimed_verified
2024-09-06T11:15:13.600Z
64c281e30344289c376c0fa4
/avatars/dae01f7054da8c9517e3808a6f10eb1a.svg
HY
user
happy0612
66dad96e75b3337a858eed2b
Sheng Jin
admin_assigned
2024-09-06T11:15:39.300Z
662e3d4e77909dd769cdc53a
/avatars/ffc32ff17aad415eef7eff6d56fe3f75.svg
ShengJin
user
ShengJin
66dad96e75b3337a858eed2c
Xiatian Zhu
admin_assigned
2024-09-06T11:15:28.374Z
647ef9e7aa8c04bbf9362a8a
/avatars/dc23e15acf7ee7f3b9f1a68c2716a66d.svg
Xiatian Zhu
user
Xiatian-Zhu
66dad96e75b3337a858eed2d
Hongxun Yao
FrozenSeg: Harmonizing Frozen Foundation Models for Open-Vocabulary
  Segmentation
Open-vocabulary segmentation poses significant challenges, as it requires
segmenting and recognizing objects across an open set of categories in
unconstrained environments. Building on the success of powerful vision-language
(ViL) foundation models, such as CLIP, recent efforts sought to harness their
zero-short capabilities to recognize unseen categories. Despite notable
performance improvements, these models still encounter the critical issue of
generating precise mask proposals for unseen categories and scenarios,
resulting in inferior segmentation performance eventually. To address this
challenge, we introduce a novel approach, FrozenSeg, designed to integrate
spatial knowledge from a localization foundation model (e.g., SAM) and semantic
knowledge extracted from a ViL model (e.g., CLIP), in a synergistic framework.
Taking the ViL model's visual encoder as the feature backbone, we inject the
space-aware feature into the learnable queries and CLIP features within the
transformer decoder. In addition, we devise a mask proposal ensemble strategy
for further improving the recall rate and mask quality. To fully exploit
pre-trained knowledge while minimizing training overhead, we freeze both
foundation models, focusing optimization efforts solely on a lightweight
transformer decoder for mask proposal generation-the performance bottleneck.
Extensive experiments demonstrate that FrozenSeg advances state-of-the-art
results across various segmentation benchmarks, trained exclusively on COCO
panoptic data, and tested in a zero-shot manner. Code is available at
https://github.com/chenxi52/FrozenSeg.
FrozenSeg combines spatial knowledge from a localization model and semantic knowledge from a vision-language model to enhance mask proposal quality in open-vocabulary segmentation.
open-vocabulary segmentation zero-shot learning CLIP SAM localization foundation model vision-language (ViL) foundation models visual encoder transformer decoder mask proposal ensemble COCO panoptic data