Efficient Speech Language Modeling via Energy Distance in Continuous
  Latent Space
We introduce SLED, an alternative approach to speech language modeling by
encoding speech waveforms into sequences of continuous latent representations
and modeling them autoregressively using an energy distance objective. The
energy distance offers an analytical measure of the distributional gap by
contrasting simulated and target samples, enabling efficient training to
capture the underlying continuous autoregressive distribution. By bypassing
reliance on residual vector quantization, SLED avoids discretization errors and
eliminates the need for the complicated hierarchical architectures common in
existing speech language models. It simplifies the overall modeling pipeline
while preserving the richness of speech information and maintaining inference
efficiency. Empirical results demonstrate that SLED achieves strong performance
in both zero-shot and streaming speech synthesis, showing its potential for
broader applications in general-purpose speech language models.
682cc6e7cb2128493cbe31b0
Zhengrui Ma
claimed_verified
2025-05-21T08:42:41.951Z
64896496ffac083138eef1e9
/avatars/216167afeb302896511deb093dbf8700.svg
Zhengrui Ma
user
Paulmzr
682cc6e7cb2128493cbe31b1
Yang Feng
682cc6e7cb2128493cbe31b2
Chenze Shao
682cc6e7cb2128493cbe31b3
Fandong Meng
682cc6e7cb2128493cbe31b4
Jie Zhou
682cc6e7cb2128493cbe31b5
Min Zhang
Efficient Speech Language Modeling via Energy Distance in Continuous
  Latent Space
We introduce SLED, an alternative approach to speech language modeling by
encoding speech waveforms into sequences of continuous latent representations
and modeling them autoregressively using an energy distance objective. The
energy distance offers an analytical measure of the distributional gap by
contrasting simulated and target samples, enabling efficient training to
capture the underlying continuous autoregressive distribution. By bypassing
reliance on residual vector quantization, SLED avoids discretization errors and
eliminates the need for the complicated hierarchical architectures common in
existing speech language models. It simplifies the overall modeling pipeline
while preserving the richness of speech information and maintaining inference
efficiency. Empirical results demonstrate that SLED achieves strong performance
in both zero-shot and streaming speech synthesis, showing its potential for
broader applications in general-purpose speech language models.
SLED encodes speech waveforms into continuous latent representations and uses an energy distance objective to model them autoregressively for efficient and accurate speech synthesis.
speech language modeling speech waveforms continuous latent representations autoregressive modeling energy distance distributional gap residual vector quantization hierarchical architectures zero-shot speech synthesis streaming speech synthesis