Open-Vocabulary Audio-Visual Semantic Segmentation
Audio-visual semantic segmentation (AVSS) aims to segment and classify
sounding objects in videos with acoustic cues. However, most approaches operate
on the close-set assumption and only identify pre-defined categories from
training data, lacking the generalization ability to detect novel categories in
practical applications. In this paper, we introduce a new task: open-vocabulary
audio-visual semantic segmentation, extending AVSS task to open-world scenarios
beyond the annotated label space. This is a more challenging task that requires
recognizing all categories, even those that have never been seen nor heard
during training. Moreover, we propose the first open-vocabulary AVSS framework,
OV-AVSS, which mainly consists of two parts: 1) a universal sound source
localization module to perform audio-visual fusion and locate all potential
sounding objects and 2) an open-vocabulary classification module to predict
categories with the help of the prior knowledge from large-scale pre-trained
vision-language models. To properly evaluate the open-vocabulary AVSS, we split
zero-shot training and testing subsets based on the AVSBench-semantic
benchmark, namely AVSBench-OV. Extensive experiments demonstrate the strong
segmentation and zero-shot generalization ability of our model on all
categories. On the AVSBench-OV dataset, OV-AVSS achieves 55.43% mIoU on base
categories and 29.14% mIoU on novel categories, exceeding the state-of-the-art
zero-shot method by 41.88%/20.61% and open-vocabulary method by 10.2%/11.6%.
The code is available at https://github.com/ruohaoguo/ovavss.
66ab2e17c5f3bec3ef2eec13
Ruohao Guo
admin_assigned
2024-08-02T15:30:32.120Z
628711eb40423ef48fbc07af
https://cdn-avatars.huggingface.co/v1/production/uploads/628711eb40423ef48fbc07af/QmW1ow6LFNxPa0TyYBAJY.jpeg
Ruohao Guo
user
ruohao
66ab2e17c5f3bec3ef2eec14
Liao Qu
66ab2e17c5f3bec3ef2eec15
Dantong Niu
admin_assigned
2024-08-02T15:31:17.561Z
65c3fdf79d062be813813e45
/avatars/52528a61abe5bbbef4a4a431944973cd.svg
Dantong Niu
user
NdtSoCool
66ab2e17c5f3bec3ef2eec16
Yanyu Qi
66ab2e17c5f3bec3ef2eec17
Wenzhen Yue
66ab2e17c5f3bec3ef2eec18
Ji Shi
66ab2e17c5f3bec3ef2eec19
Bowei Xing
66ab2e17c5f3bec3ef2eec1a
Xianghua Ying
Open-Vocabulary Audio-Visual Semantic Segmentation
Audio-visual semantic segmentation (AVSS) aims to segment and classify
sounding objects in videos with acoustic cues. However, most approaches operate
on the close-set assumption and only identify pre-defined categories from
training data, lacking the generalization ability to detect novel categories in
practical applications. In this paper, we introduce a new task: open-vocabulary
audio-visual semantic segmentation, extending AVSS task to open-world scenarios
beyond the annotated label space. This is a more challenging task that requires
recognizing all categories, even those that have never been seen nor heard
during training. Moreover, we propose the first open-vocabulary AVSS framework,
OV-AVSS, which mainly consists of two parts: 1) a universal sound source
localization module to perform audio-visual fusion and locate all potential
sounding objects and 2) an open-vocabulary classification module to predict
categories with the help of the prior knowledge from large-scale pre-trained
vision-language models. To properly evaluate the open-vocabulary AVSS, we split
zero-shot training and testing subsets based on the AVSBench-semantic
benchmark, namely AVSBench-OV. Extensive experiments demonstrate the strong
segmentation and zero-shot generalization ability of our model on all
categories. On the AVSBench-OV dataset, OV-AVSS achieves 55.43% mIoU on base
categories and 29.14% mIoU on novel categories, exceeding the state-of-the-art
zero-shot method by 41.88%/20.61% and open-vocabulary method by 10.2%/11.6%.
The code is available at https://github.com/ruohaoguo/ovavss.
OV-AVSS, a new framework for open-vocabulary audio-visual semantic segmentation, achieves strong performance across both base and novel categories by combining sound source localization and open-vocabulary classification.
audio-visual semantic segmentation open-vocabulary universal sound source localization audio-visual fusion open-vocabulary classification vision-language models zero-shot generalization mIoU AVSBench-semantic AVSBench-OV