SoundCTM: Uniting Score-based and Consistency Models for Text-to-Sound
  Generation
Sound content is an indispensable element for multimedia works such as video
games, music, and films. Recent high-quality diffusion-based sound generation
models can serve as valuable tools for the creators. However, despite producing
high-quality sounds, these models often suffer from slow inference speeds. This
drawback burdens creators, who typically refine their sounds through trial and
error to align them with their artistic intentions. To address this issue, we
introduce Sound Consistency Trajectory Models (SoundCTM). Our model enables
flexible transitioning between high-quality 1-step sound generation and
superior sound quality through multi-step generation. This allows creators to
initially control sounds with 1-step samples before refining them through
multi-step generation. While CTM fundamentally achieves flexible 1-step and
multi-step generation, its impressive performance heavily depends on an
additional pretrained feature extractor and an adversarial loss, which are
expensive to train and not always available in other domains. Thus, we reframe
CTM's training framework and introduce a novel feature distance by utilizing
the teacher's network for a distillation loss. Additionally, while distilling
classifier-free guided trajectories, we train conditional and unconditional
student models simultaneously and interpolate between these models during
inference. We also propose training-free controllable frameworks for SoundCTM,
leveraging its flexible sampling capability. SoundCTM achieves both promising
1-step and multi-step real-time sound generation without using any extra
off-the-shelf networks. Furthermore, we demonstrate SoundCTM's capability of
controllable sound generation in a training-free manner.
66588e3daad054f64123df03
Koichi Saito
extracted_pending
2024-06-04T19:56:16.447Z
665f7129b75fa1b1a705b7ce
/avatars/f9c8245c0441dc428edcefce8989444e.svg
Koichi Saito
user
koichisaito
66588e3daad054f64123df04
Dongjun Kim
claimed_verified
2024-10-18T08:27:31.977Z
65dc46602bc96e6f0062e936
/avatars/3d4d68830fafd35f7d3769b58149ff34.svg
Dongjun Kim
user
partitionsofunity
66588e3daad054f64123df05
Takashi Shibuya
admin_assigned
2024-05-31T11:14:49.051Z
6650773ca6acfdd2aba7d486
/avatars/d297886ea60dbff98a043caf825820ed.svg
Takashi Shibuya
user
TakashiShibuyaSony
66588e3daad054f64123df06
Chieh-Hsin Lai
66588e3daad054f64123df07
Zhi Zhong
66588e3daad054f64123df08
Yuhta Takida
admin_assigned
2024-05-31T11:15:08.191Z
66138c4074f830bc7d9d6622
/avatars/d50da6d7597d3bcf63f9f0c74e910155.svg
Yuhta Takida
user
ytakida
66588e3daad054f64123df09
Yuki Mitsufuji
SoundCTM: Uniting Score-based and Consistency Models for Text-to-Sound
  Generation
Sound content is an indispensable element for multimedia works such as video
games, music, and films. Recent high-quality diffusion-based sound generation
models can serve as valuable tools for the creators. However, despite producing
high-quality sounds, these models often suffer from slow inference speeds. This
drawback burdens creators, who typically refine their sounds through trial and
error to align them with their artistic intentions. To address this issue, we
introduce Sound Consistency Trajectory Models (SoundCTM). Our model enables
flexible transitioning between high-quality 1-step sound generation and
superior sound quality through multi-step generation. This allows creators to
initially control sounds with 1-step samples before refining them through
multi-step generation. While CTM fundamentally achieves flexible 1-step and
multi-step generation, its impressive performance heavily depends on an
additional pretrained feature extractor and an adversarial loss, which are
expensive to train and not always available in other domains. Thus, we reframe
CTM's training framework and introduce a novel feature distance by utilizing
the teacher's network for a distillation loss. Additionally, while distilling
classifier-free guided trajectories, we train conditional and unconditional
student models simultaneously and interpolate between these models during
inference. We also propose training-free controllable frameworks for SoundCTM,
leveraging its flexible sampling capability. SoundCTM achieves both promising
1-step and multi-step real-time sound generation without using any extra
off-the-shelf networks. Furthermore, we demonstrate SoundCTM's capability of
controllable sound generation in a training-free manner.
SoundCTM efficiently generates high-quality sounds in real-time using both 1-step and multi-step methods, employing a teacher-student framework and controllable sound generation without additional networks.
diffusion-based sound generation Sound Consistency Trajectory Models (SoundCTM) 1-step sound generation multi-step generation pretrained feature extractor adversarial loss distillation loss conditional models unconditional models feature distance classifier-free guided trajectories training-free controllable frameworks