Efficient Audio Captioning with Encoder-Level Knowledge Distillation
Significant improvement has been achieved in automated audio captioning (AAC)
with recent models. However, these models have become increasingly large as
their performance is enhanced. In this work, we propose a knowledge
distillation (KD) framework for AAC. Our analysis shows that in the
encoder-decoder based AAC models, it is more effective to distill knowledge
into the encoder as compared with the decoder. To this end, we incorporate
encoder-level KD loss into training, in addition to the standard supervised
loss and sequence-level KD loss. We investigate two encoder-level KD methods,
based on mean squared error (MSE) loss and contrastive loss, respectively.
Experimental results demonstrate that contrastive KD is more robust than MSE
KD, exhibiting superior performance in data-scarce situations. By leveraging
audio-only data into training in the KD framework, our student model achieves
competitive performance, with an inference speed that is 19 times
fasterAn online demo is available at
\url{https://huggingface.co/spaces/wsntxxn/efficient_audio_captioning}.
669dbdc754725cd6e0051485
Xuenan Xu
admin_assigned
2024-07-22T12:50:30.406Z
6457623c90bf61c87f9ffdc4
https://cdn-avatars.huggingface.co/v1/production/uploads/6457623c90bf61c87f9ffdc4/pKG76YipWfYFhSnorepsl.jpeg
Xuenan Xu
user
wsntxxn
669dbdc754725cd6e0051486
Haohe Liu
admin_assigned
2024-07-22T12:50:38.888Z
6155245d1c762d4d61b51d5d
https://cdn-avatars.huggingface.co/v1/production/uploads/1677233708065-6155245d1c762d4d61b51d5d.png
haoheliu
user
haoheliu
669dbdc754725cd6e0051487
Mengyue Wu
669dbdc754725cd6e0051488
Wenwu Wang
admin_assigned
2024-07-22T12:50:58.168Z
65c30ec23d8075e46a3da733
/avatars/164d4da7ba66f451ed08f45b76a42ffe.svg
wangwenwu
user
wangwenwu2
669dbdc754725cd6e0051489
Mark D. Plumbley
Efficient Audio Captioning with Encoder-Level Knowledge Distillation
Significant improvement has been achieved in automated audio captioning (AAC)
with recent models. However, these models have become increasingly large as
their performance is enhanced. In this work, we propose a knowledge
distillation (KD) framework for AAC. Our analysis shows that in the
encoder-decoder based AAC models, it is more effective to distill knowledge
into the encoder as compared with the decoder. To this end, we incorporate
encoder-level KD loss into training, in addition to the standard supervised
loss and sequence-level KD loss. We investigate two encoder-level KD methods,
based on mean squared error (MSE) loss and contrastive loss, respectively.
Experimental results demonstrate that contrastive KD is more robust than MSE
KD, exhibiting superior performance in data-scarce situations. By leveraging
audio-only data into training in the KD framework, our student model achieves
competitive performance, with an inference speed that is 19 times
fasterAn online demo is available at
\url{https://huggingface.co/spaces/wsntxxn/efficient_audio_captioning}.
A knowledge distillation framework for automated audio captioning improves performance and speed by focusing on distilling knowledge into the encoder, with contrastive KD showing particular effectiveness in data-scarce settings.
knowledge distillation kd framework encoder-decoder encoder-level kd mse loss contrastive loss automated audio captioning audio-only data competitive performance inference speed