HiFi-SR: A Unified Generative Transformer-Convolutional Adversarial
  Network for High-Fidelity Speech Super-Resolution
The application of generative adversarial networks (GANs) has recently
advanced speech super-resolution (SR) based on intermediate representations
like mel-spectrograms. However, existing SR methods that typically rely on
independently trained and concatenated networks may lead to inconsistent
representations and poor speech quality, especially in out-of-domain scenarios.
In this work, we propose HiFi-SR, a unified network that leverages end-to-end
adversarial training to achieve high-fidelity speech super-resolution. Our
model features a unified transformer-convolutional generator designed to
seamlessly handle both the prediction of latent representations and their
conversion into time-domain waveforms. The transformer network serves as a
powerful encoder, converting low-resolution mel-spectrograms into latent space
representations, while the convolutional network upscales these representations
into high-resolution waveforms. To enhance high-frequency fidelity, we
incorporate a multi-band, multi-scale time-frequency discriminator, along with
a multi-scale mel-reconstruction loss in the adversarial training process.
HiFi-SR is versatile, capable of upscaling any input speech signal between 4
kHz and 32 kHz to a 48 kHz sampling rate. Experimental results demonstrate that
HiFi-SR significantly outperforms existing speech SR methods across both
objective metrics and ABX preference tests, for both in-domain and
out-of-domain scenarios (https://github.com/modelscope/ClearerVoice-Studio).
678dd3044ce7abd7ef1f5345
Shengkui Zhao
678dd3044ce7abd7ef1f5346
Kun Zhou
678dd3044ce7abd7ef1f5347
Zexu Pan
678dd3044ce7abd7ef1f5348
Yukun Ma
678dd3044ce7abd7ef1f5349
Chong Zhang
claimed_verified
2025-03-18T08:12:20.391Z
649aa7d0ef592b26c3d4ead4
/avatars/1174fe2a396b553312896914401947cb.svg
iris2c
user
iris2c
678dd3044ce7abd7ef1f534a
Bin Ma
HiFi-SR: A Unified Generative Transformer-Convolutional Adversarial
  Network for High-Fidelity Speech Super-Resolution
The application of generative adversarial networks (GANs) has recently
advanced speech super-resolution (SR) based on intermediate representations
like mel-spectrograms. However, existing SR methods that typically rely on
independently trained and concatenated networks may lead to inconsistent
representations and poor speech quality, especially in out-of-domain scenarios.
In this work, we propose HiFi-SR, a unified network that leverages end-to-end
adversarial training to achieve high-fidelity speech super-resolution. Our
model features a unified transformer-convolutional generator designed to
seamlessly handle both the prediction of latent representations and their
conversion into time-domain waveforms. The transformer network serves as a
powerful encoder, converting low-resolution mel-spectrograms into latent space
representations, while the convolutional network upscales these representations
into high-resolution waveforms. To enhance high-frequency fidelity, we
incorporate a multi-band, multi-scale time-frequency discriminator, along with
a multi-scale mel-reconstruction loss in the adversarial training process.
HiFi-SR is versatile, capable of upscaling any input speech signal between 4
kHz and 32 kHz to a 48 kHz sampling rate. Experimental results demonstrate that
HiFi-SR significantly outperforms existing speech SR methods across both
objective metrics and ABX preference tests, for both in-domain and
out-of-domain scenarios (https://github.com/modelscope/ClearerVoice-Studio).
HiFi-SR, a unified GAN-based model, achieves high-fidelity speech super-resolution using a transformer-convolutional generator and multi-scale discriminators, outperforming existing methods across various scenarios.
generative adversarial networks GANs speech super-resolution SR mel-spectrograms unified network end-to-end adversarial training transformer-convolutional generator latent space representations time-domain waveforms multi-band multi-scale time-frequency discriminator multi-scale mel-reconstruction loss