DITTO: Diffusion Inference-Time T-Optimization for Music Generation
We propose Diffusion Inference-Time T-Optimization (DITTO), a general-purpose
frame-work for controlling pre-trained text-to-music diffusion models at
inference-time via optimizing initial noise latents. Our method can be used to
optimize through any differentiable feature matching loss to achieve a target
(stylized) output and leverages gradient checkpointing for memory efficiency.
We demonstrate a surprisingly wide-range of applications for music generation
including inpainting, outpainting, and looping as well as intensity, melody,
and musical structure control - all without ever fine-tuning the underlying
model. When we compare our approach against related training, guidance, and
optimization-based methods, we find DITTO achieves state-of-the-art performance
on nearly all tasks, including outperforming comparable approaches on
controllability, audio quality, and computational efficiency, thus opening the
door for high-quality, flexible, training-free control of diffusion models.
Sound examples can be found at https://DITTO-Music.github.io/web/.
65af528552e1b2aae42443fb
Zachary Novack
extracted_confirmed
2024-01-23T05:47:52.680Z
643060c6cb3fe707b24c53a2
https://cdn-avatars.huggingface.co/v1/production/uploads/643060c6cb3fe707b24c53a2/MIoM9hrX0vV4XRyrm-4Kz.jpeg
Zachary Novack
user
ZacharyNovack
65af528552e1b2aae42443fc
Julian McAuley
65af528552e1b2aae42443fd
Taylor Berg-Kirkpatrick
65af528552e1b2aae42443fe
Nicholas J. Bryan
claimed_verified
2024-01-23T06:35:21.161Z
648119ee7a741c7f33f49f25
/avatars/183d2143ba20e1cc8712c63c055aadd7.svg
Nicholas J. Bryan
user
Njb
DITTO: Diffusion Inference-Time T-Optimization for Music Generation
We propose Diffusion Inference-Time T-Optimization (DITTO), a general-purpose
frame-work for controlling pre-trained text-to-music diffusion models at
inference-time via optimizing initial noise latents. Our method can be used to
optimize through any differentiable feature matching loss to achieve a target
(stylized) output and leverages gradient checkpointing for memory efficiency.
We demonstrate a surprisingly wide-range of applications for music generation
including inpainting, outpainting, and looping as well as intensity, melody,
and musical structure control - all without ever fine-tuning the underlying
model. When we compare our approach against related training, guidance, and
optimization-based methods, we find DITTO achieves state-of-the-art performance
on nearly all tasks, including outperforming comparable approaches on
controllability, audio quality, and computational efficiency, thus opening the
door for high-quality, flexible, training-free control of diffusion models.
Sound examples can be found at https://DITTO-Music.github.io/web/.
DITTO optimizes noise latents during inference to control text-to-music diffusion models, achieving superior performance in various tasks without fine-tuning.
Diffusion Inference-Time T-Optimization pre-trained text-to-music diffusion models initial noise latents differentiable feature matching loss gradient checkpointing music generation inpainting outpainting looping intensity control melody control musical structure control training-free control