ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds
Open-vocabulary audio-language models, like CLAP, offer a promising approach
for zero-shot audio classification (ZSAC) by enabling classification with any
arbitrary set of categories specified with natural language prompts. In this
paper, we propose a simple but effective method to improve ZSAC with CLAP.
Specifically, we shift from the conventional method of using prompts with
abstract category labels (e.g., Sound of an organ) to prompts that describe
sounds using their inherent descriptive features in a diverse context (e.g.,The
organ's deep and resonant tones filled the cathedral.). To achieve this, we
first propose ReCLAP, a CLAP model trained with rewritten audio captions for
improved understanding of sounds in the wild. These rewritten captions describe
each sound event in the original caption using their unique discriminative
characteristics. ReCLAP outperforms all baselines on both multi-modal
audio-text retrieval and ZSAC. Next, to improve zero-shot audio classification
with ReCLAP, we propose prompt augmentation. In contrast to the traditional
method of employing hand-written template prompts, we generate custom prompts
for each unique label in the dataset. These custom prompts first describe the
sound event in the label and then employ them in diverse scenes. Our proposed
method improves ReCLAP's performance on ZSAC by 1%-18% and outperforms all
baselines by 1% - 55%.
66e8ee1ee39084cfc7f233c9
Sreyan Ghosh
claimed_verified
2024-10-28T15:36:37.734Z
62c9664eb34e600d7eaa4beb
/avatars/ca23ecdec2d31c99ecce97d9b180ae0c.svg
Ghosh
user
Sreyan88
66e8ee1ee39084cfc7f233ca
Sonal Kumar
claimed_verified
2024-09-17T15:08:14.641Z
65203f31d6dec04046139874
https://cdn-avatars.huggingface.co/v1/production/uploads/65203f31d6dec04046139874/1LDWW4KblYeIDve9F_kUA.png
Sonal Kumar
user
sonalkum
66e8ee1ee39084cfc7f233cb
Chandra Kiran Reddy Evuru
66e8ee1ee39084cfc7f233cc
Oriol Nieto
admin_assigned
2024-09-17T07:50:57.540Z
657a29d7d3e458405b8dc146
/avatars/1fe425dcf3133462fb6376e8e1550b4a.svg
Oriol Nieto
user
urinieto
66e8ee1ee39084cfc7f233cd
Ramani Duraiswami
admin_assigned
2024-09-17T07:50:51.405Z
65b8f0339b7250e205100190
/avatars/29df91154c5911a44ef503bb3580ac81.svg
Ramani Duraiswami
user
RamaniD
66e8ee1ee39084cfc7f233ce
Dinesh Manocha
admin_assigned
2024-09-17T07:50:44.680Z
6537a569568d8be8fa096b8c
/avatars/bfda5cb252d8b5bc3ad737d99c0d7f49.svg
Dinesh Manocha
user
manocha
ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds
Open-vocabulary audio-language models, like CLAP, offer a promising approach
for zero-shot audio classification (ZSAC) by enabling classification with any
arbitrary set of categories specified with natural language prompts. In this
paper, we propose a simple but effective method to improve ZSAC with CLAP.
Specifically, we shift from the conventional method of using prompts with
abstract category labels (e.g., Sound of an organ) to prompts that describe
sounds using their inherent descriptive features in a diverse context (e.g.,The
organ's deep and resonant tones filled the cathedral.). To achieve this, we
first propose ReCLAP, a CLAP model trained with rewritten audio captions for
improved understanding of sounds in the wild. These rewritten captions describe
each sound event in the original caption using their unique discriminative
characteristics. ReCLAP outperforms all baselines on both multi-modal
audio-text retrieval and ZSAC. Next, to improve zero-shot audio classification
with ReCLAP, we propose prompt augmentation. In contrast to the traditional
method of employing hand-written template prompts, we generate custom prompts
for each unique label in the dataset. These custom prompts first describe the
sound event in the label and then employ them in diverse scenes. Our proposed
method improves ReCLAP's performance on ZSAC by 1%-18% and outperforms all
baselines by 1% - 55%.
A method employing ReCLAP and prompt augmentation enhances zero-shot audio classification by using descriptive audio features in context-specific prompts.
CLAP zero-shot audio classification ZSAC ReCLAP multi-modal audio-text retrieval prompt augmentation