VampNet: Music Generation via Masked Acoustic Token Modeling
We introduce VampNet, a masked acoustic token modeling approach to music
synthesis, compression, inpainting, and variation. We use a variable masking
schedule during training which allows us to sample coherent music from the
model by applying a variety of masking approaches (called prompts) during
inference. VampNet is non-autoregressive, leveraging a bidirectional
transformer architecture that attends to all tokens in a forward pass. With
just 36 sampling passes, VampNet can generate coherent high-fidelity musical
waveforms. We show that by prompting VampNet in various ways, we can apply it
to tasks like music compression, inpainting, outpainting, continuation, and
looping with variation (vamping). Appropriately prompted, VampNet is capable of
maintaining style, genre, instrumentation, and other high-level aspects of the
music. This flexible prompting capability makes VampNet a powerful music
co-creation tool. Code and audio samples are available online.
64acc3f1f5d40283c620a0fe
Hugo Flores Garcia
claimed_verified
2023-07-12T07:29:30.137Z
60d3a674b8448e1785bbda24
https://cdn-avatars.huggingface.co/v1/production/uploads/60d3a674b8448e1785bbda24/8BS8q0VqI6baD0hRMSg6T.jpeg
Hugo Flores
user
hugggof
64acc3f1f5d40283c620a0ff
Prem Seetharaman
64acc3f1f5d40283c620a100
Rithesh Kumar
64acc3f1f5d40283c620a101
Bryan Pardo
VampNet: Music Generation via Masked Acoustic Token Modeling
We introduce VampNet, a masked acoustic token modeling approach to music
synthesis, compression, inpainting, and variation. We use a variable masking
schedule during training which allows us to sample coherent music from the
model by applying a variety of masking approaches (called prompts) during
inference. VampNet is non-autoregressive, leveraging a bidirectional
transformer architecture that attends to all tokens in a forward pass. With
just 36 sampling passes, VampNet can generate coherent high-fidelity musical
waveforms. We show that by prompting VampNet in various ways, we can apply it
to tasks like music compression, inpainting, outpainting, continuation, and
looping with variation (vamping). Appropriately prompted, VampNet is capable of
maintaining style, genre, instrumentation, and other high-level aspects of the
music. This flexible prompting capability makes VampNet a powerful music
co-creation tool. Code and audio samples are available online.
VampNet, a non-autoregressive transformer-based model, synthesizes and manipulates music with high-fidelity using variable masking and prompting techniques.
masked acoustic token modeling music synthesis music compression music inpainting bidirectional transformer architecture prompting high-fidelity musical waveforms music co-creation