DMOSpeech 2: Reinforcement Learning for Duration Prediction in
  Metric-Optimized Speech Synthesis
Diffusion-based text-to-speech (TTS) systems have made remarkable progress in
zero-shot speech synthesis, yet optimizing all components for perceptual
metrics remains challenging. Prior work with DMOSpeech demonstrated direct
metric optimization for speech generation components, but duration prediction
remained unoptimized. This paper presents DMOSpeech 2, which extends metric
optimization to the duration predictor through a reinforcement learning
approach. The proposed system implements a novel duration policy framework
using group relative preference optimization (GRPO) with speaker similarity and
word error rate as reward signals. By optimizing this previously unoptimized
component, DMOSpeech 2 creates a more complete metric-optimized synthesis
pipeline. Additionally, this paper introduces teacher-guided sampling, a hybrid
approach leveraging a teacher model for initial denoising steps before
transitioning to the student model, significantly improving output diversity
while maintaining efficiency. Comprehensive evaluations demonstrate superior
performance across all metrics compared to previous systems, while reducing
sampling steps by half without quality degradation. These advances represent a
significant step toward speech synthesis systems with metric optimization
across multiple components. The audio samples, code and pre-trained models are
available at https://dmospeech2.github.io/.
6882ea91846d2e78b7548a49
Yinghao Aaron Li
6882ea91846d2e78b7548a4a
Xilin Jiang
6882ea91846d2e78b7548a4b
Fei Tao
6882ea91846d2e78b7548a4c
Cheng Niu
6882ea91846d2e78b7548a4d
Kaifeng Xu
6882ea91846d2e78b7548a4e
Juntong Song
6882ea91846d2e78b7548a4f
Nima Mesgarani
DMOSpeech 2: Reinforcement Learning for Duration Prediction in
  Metric-Optimized Speech Synthesis
Diffusion-based text-to-speech (TTS) systems have made remarkable progress in
zero-shot speech synthesis, yet optimizing all components for perceptual
metrics remains challenging. Prior work with DMOSpeech demonstrated direct
metric optimization for speech generation components, but duration prediction
remained unoptimized. This paper presents DMOSpeech 2, which extends metric
optimization to the duration predictor through a reinforcement learning
approach. The proposed system implements a novel duration policy framework
using group relative preference optimization (GRPO) with speaker similarity and
word error rate as reward signals. By optimizing this previously unoptimized
component, DMOSpeech 2 creates a more complete metric-optimized synthesis
pipeline. Additionally, this paper introduces teacher-guided sampling, a hybrid
approach leveraging a teacher model for initial denoising steps before
transitioning to the student model, significantly improving output diversity
while maintaining efficiency. Comprehensive evaluations demonstrate superior
performance across all metrics compared to previous systems, while reducing
sampling steps by half without quality degradation. These advances represent a
significant step toward speech synthesis systems with metric optimization
across multiple components. The audio samples, code and pre-trained models are
available at https://dmospeech2.github.io/.
DMOSpeech 2 optimizes duration prediction in diffusion-based text-to-speech using reinforcement learning and introduces teacher-guided sampling to enhance diversity and efficiency.
diffusion-based text-to-speech DMOSpeech duration prediction reinforcement learning group relative preference optimization speaker similarity word error rate teacher-guided sampling speech synthesis metric optimization