Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space
Zhengrui Ma, Yang Feng, Chenze Shao, Fandong Meng, Jie Zhou, Min Zhang
We introduce SLED, an alternative approach to speech language modeling by encoding speech waveforms into continuous latent representation sequences and modeling them autoregressively using an energy distance objective in this continuous space. By bypassing the reliance on residual vector quantization, it avoids discretization errors and eliminates the need for the complicated hierarchical architectures common in existing speech language models. It simplifies the overall modeling pipeline while preserving the richness of speech information and maintaining inference efficiency. Empirical results demonstrate that SLED achieves strong performance in both zero-shot and streaming speech synthesis, showing its potential for broader applications in general-purpose speech language modeling.