Enhance audio generation controllability through representation
  similarity regularization
This paper presents an innovative approach to enhance control over audio
generation by emphasizing the alignment between audio and text representations
during model training. In the context of language model-based audio generation,
the model leverages input from both textual and audio token representations to
predict subsequent audio tokens. However, the current configuration lacks
explicit regularization to ensure the alignment between the chosen text
representation and the language model's predictions. Our proposal involves the
incorporation of audio and text representation regularization, particularly
during the classifier-free guidance (CFG) phase, where the text condition is
excluded from cross attention during language model training. The aim of this
proposed representation regularization is to minimize discrepancies in audio
and text similarity compared to other samples within the same training batch.
Experimental results on both music and audio generation tasks demonstrate that
our proposed methods lead to improvements in objective metrics for both audio
and music generation, as well as an enhancement in the human perception for
audio generation.
65090b74b44445e9b378e19f
Yangyang Shi
65090b74b44445e9b378e1a0
Gael Le Lan
65090b74b44445e9b378e1a1
Varun Nagaraja
65090b74b44445e9b378e1a2
Zhaoheng Ni
65090b74b44445e9b378e1a3
Xinhao Mei
65090b74b44445e9b378e1a4
Ernie Chang
65090b74b44445e9b378e1a5
Forrest Iandola
65090b74b44445e9b378e1a6
Yang Liu
65090b74b44445e9b378e1a7
Vikas Chandra
Enhance audio generation controllability through representation
  similarity regularization
This paper presents an innovative approach to enhance control over audio
generation by emphasizing the alignment between audio and text representations
during model training. In the context of language model-based audio generation,
the model leverages input from both textual and audio token representations to
predict subsequent audio tokens. However, the current configuration lacks
explicit regularization to ensure the alignment between the chosen text
representation and the language model's predictions. Our proposal involves the
incorporation of audio and text representation regularization, particularly
during the classifier-free guidance (CFG) phase, where the text condition is
excluded from cross attention during language model training. The aim of this
proposed representation regularization is to minimize discrepancies in audio
and text similarity compared to other samples within the same training batch.
Experimental results on both music and audio generation tasks demonstrate that
our proposed methods lead to improvements in objective metrics for both audio
and music generation, as well as an enhancement in the human perception for
audio generation.
The paper enhances audio generation by introducing audio and text representation regularization during the classifier-free guidance phase of language model training, improving similarity alignment and perceptual quality.
language model audio generation textual tokens audio tokens cross-attention classifier-free guidance representation regularization