Stack-and-Delay: a new codebook pattern for music generation
In language modeling based music generation, a generated waveform is
represented by a sequence of hierarchical token stacks that can be decoded
either in an auto-regressive manner or in parallel, depending on the codebook
patterns. In particular, flattening the codebooks represents the highest
quality decoding strategy, while being notoriously slow. To this end, we
propose a novel stack-and-delay style of decoding strategy to improve upon the
flat pattern decoding where generation speed is four times faster as opposed to
vanilla flat decoding. This brings the inference time close to that of the
delay decoding strategy, and allows for faster inference on GPU for small batch
sizes. For the same inference efficiency budget as the delay pattern, we show
that the proposed approach performs better in objective evaluations, almost
closing the gap with the flat pattern in terms of quality. The results are
corroborated by subjective evaluations which show that samples generated by the
new model are slightly more often preferred to samples generated by the
competing model given the same text prompts.
6509155c7b68c4a6f65f3f3b
Gael Le Lan
6509155c7b68c4a6f65f3f3c
Varun Nagaraja
6509155c7b68c4a6f65f3f3d
Ernie Chang
6509155c7b68c4a6f65f3f3e
David Kant
6509155c7b68c4a6f65f3f3f
Zhaoheng Ni
6509155c7b68c4a6f65f3f40
Yangyang Shi
6509155c7b68c4a6f65f3f41
Forrest Iandola
6509155c7b68c4a6f65f3f42
Vikas Chandra
Stack-and-Delay: a new codebook pattern for music generation
In language modeling based music generation, a generated waveform is
represented by a sequence of hierarchical token stacks that can be decoded
either in an auto-regressive manner or in parallel, depending on the codebook
patterns. In particular, flattening the codebooks represents the highest
quality decoding strategy, while being notoriously slow. To this end, we
propose a novel stack-and-delay style of decoding strategy to improve upon the
flat pattern decoding where generation speed is four times faster as opposed to
vanilla flat decoding. This brings the inference time close to that of the
delay decoding strategy, and allows for faster inference on GPU for small batch
sizes. For the same inference efficiency budget as the delay pattern, we show
that the proposed approach performs better in objective evaluations, almost
closing the gap with the flat pattern in terms of quality. The results are
corroborated by subjective evaluations which show that samples generated by the
new model are slightly more often preferred to samples generated by the
competing model given the same text prompts.
A new stack-and-delay decoding strategy for hierarchical token stacks in music generation improves inference speed without significantly compromising quality.
language modeling hierarchical token stacks auto-regressive parallel decoding codebook stack-and-delay inference time GPU batch sizes objective evaluations subjective evaluations