Going Denser with Open-Vocabulary Part Segmentation
Object detection has been expanded from a limited number of categories to
open vocabulary. Moving forward, a complete intelligent vision system requires
understanding more fine-grained object descriptions, object parts. In this
paper, we propose a detector with the ability to predict both open-vocabulary
objects and their part segmentation. This ability comes from two designs.
First, we train the detector on the joint of part-level, object-level and
image-level data to build the multi-granularity alignment between language and
image. Second, we parse the novel object into its parts by its dense semantic
correspondence with the base object. These two designs enable the detector to
largely benefit from various data sources and foundation models. In
open-vocabulary part segmentation experiments, our method outperforms the
baseline by 3.3sim7.3 mAP in cross-dataset generalization on PartImageNet,
and improves the baseline by 7.3 novel AP_{50} in cross-category
generalization on Pascal Part. Finally, we train a detector that generalizes to
a wide range of part segmentation datasets while achieving better performance
than dataset-specific training.
646705fdcdea48159aa5e576
Peize Sun
claimed_verified
2023-05-19T19:04:05.378Z
640dc9bf8512ec51d7f0ac1a
https://cdn-avatars.huggingface.co/v1/production/uploads/640dc9bf8512ec51d7f0ac1a/sT4rdEoQbzfW6D3xDVdqt.jpeg
peizesun
user
peizesun
646705fdcdea48159aa5e577
Shoufa Chen
admin_assigned
2023-05-23T09:50:21.566Z
6412a33900634c4fe9873652
https://cdn-avatars.huggingface.co/v1/production/uploads/6412a33900634c4fe9873652/Nmn_yRA1gGD2VO1YbSOYF.jpeg
Shoufa Chen
user
ShoufaChen
646705fdcdea48159aa5e578
Chenchen Zhu
646705fdcdea48159aa5e579
Fanyi Xiao
admin_assigned
2023-05-23T09:50:59.370Z
641cf83645487810d136cb34
/avatars/95bfc200f86abbe9e8df0308517a9ce4.svg
Fanyi Xiao
user
fanyix
646705fdcdea48159aa5e57a
Ping Luo
646705fdcdea48159aa5e57b
Saining Xie
646705fdcdea48159aa5e57c
Zhicheng Yan
Going Denser with Open-Vocabulary Part Segmentation
Object detection has been expanded from a limited number of categories to
open vocabulary. Moving forward, a complete intelligent vision system requires
understanding more fine-grained object descriptions, object parts. In this
paper, we propose a detector with the ability to predict both open-vocabulary
objects and their part segmentation. This ability comes from two designs.
First, we train the detector on the joint of part-level, object-level and
image-level data to build the multi-granularity alignment between language and
image. Second, we parse the novel object into its parts by its dense semantic
correspondence with the base object. These two designs enable the detector to
largely benefit from various data sources and foundation models. In
open-vocabulary part segmentation experiments, our method outperforms the
baseline by 3.3sim7.3 mAP in cross-dataset generalization on PartImageNet,
and improves the baseline by 7.3 novel AP_{50} in cross-category
generalization on Pascal Part. Finally, we train a detector that generalizes to
a wide range of part segmentation datasets while achieving better performance
than dataset-specific training.
A detector that predicts open-vocabulary objects and their parts using multi-granularity alignment outperforms baselines in cross-dataset and cross-category generalization.
open vocabulary object detection part segmentation multi-granularity alignment dense semantic correspondence cross-dataset generalization cross-category generalization dataset-specific training mAP novel AP$_{50}$ PartImageNet Pascal Part