Voila: Voice-Language Foundation Models for Real-Time Autonomous
  Interaction and Voice Role-Play
A voice AI agent that blends seamlessly into daily life would interact with
humans in an autonomous, real-time, and emotionally expressive manner. Rather
than merely reacting to commands, it would continuously listen, reason, and
respond proactively, fostering fluid, dynamic, and emotionally resonant
interactions. We introduce Voila, a family of large voice-language foundation
models that make a step towards this vision. Voila moves beyond traditional
pipeline systems by adopting a new end-to-end architecture that enables
full-duplex, low-latency conversations while preserving rich vocal nuances such
as tone, rhythm, and emotion. It achieves a response latency of just 195
milliseconds, surpassing the average human response time. Its hierarchical
multi-scale Transformer integrates the reasoning capabilities of large language
models (LLMs) with powerful acoustic modeling, enabling natural, persona-aware
voice generation -- where users can simply write text instructions to define
the speaker's identity, tone, and other characteristics. Moreover, Voila
supports over one million pre-built voices and efficient customization of new
ones from brief audio samples as short as 10 seconds. Beyond spoken dialogue,
Voila is designed as a unified model for a wide range of voice-based
applications, including automatic speech recognition (ASR), Text-to-Speech
(TTS), and, with minimal adaptation, multilingual speech translation. Voila is
fully open-sourced to support open research and accelerate progress toward
next-generation human-machine interactions.
6819982f17007d963b9d4166
Yemin Shi
6819982f17007d963b9d4167
Yu Shu
claimed_verified
2025-05-12T18:34:56.108Z
670f584caddfb18348a52240
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/rJCWRGII2-Kn6nc1-v44J.jpeg
YuShu
user
TabbbSY
6819982f17007d963b9d4168
Siwei Dong
6819982f17007d963b9d4169
Guangyi Liu
admin_assigned
2025-05-06T08:46:52.667Z
6108ae87823007eaf0c7bd1e
https://cdn-avatars.huggingface.co/v1/production/uploads/6108ae87823007eaf0c7bd1e/dKjdx9I5waJs6oUQ0_mmT.png
Guangyi Liu
user
guangyil
6819982f17007d963b9d416a
Jaward Sesay
claimed_verified
2025-05-06T08:32:48.746Z
6438a9027de34e8ea7e4b257
https://cdn-avatars.huggingface.co/v1/production/uploads/6438a9027de34e8ea7e4b257/vib8QSd1AWMr_bR9ig_xJ.jpeg
Jaward Sesay
user
Jaward
6819982f17007d963b9d416b
Jingwen Li
6819982f17007d963b9d416c
Zhiting Hu
admin_assigned
2025-05-06T08:46:15.191Z
665bfa1b0d71762b8613282d
/avatars/edbde7b1b47032339a1ecc59f8ea8f1a.svg
Zhiting Hu
user
zhitinghu
Voila: Voice-Language Foundation Models for Real-Time Autonomous
  Interaction and Voice Role-Play
A voice AI agent that blends seamlessly into daily life would interact with
humans in an autonomous, real-time, and emotionally expressive manner. Rather
than merely reacting to commands, it would continuously listen, reason, and
respond proactively, fostering fluid, dynamic, and emotionally resonant
interactions. We introduce Voila, a family of large voice-language foundation
models that make a step towards this vision. Voila moves beyond traditional
pipeline systems by adopting a new end-to-end architecture that enables
full-duplex, low-latency conversations while preserving rich vocal nuances such
as tone, rhythm, and emotion. It achieves a response latency of just 195
milliseconds, surpassing the average human response time. Its hierarchical
multi-scale Transformer integrates the reasoning capabilities of large language
models (LLMs) with powerful acoustic modeling, enabling natural, persona-aware
voice generation -- where users can simply write text instructions to define
the speaker's identity, tone, and other characteristics. Moreover, Voila
supports over one million pre-built voices and efficient customization of new
ones from brief audio samples as short as 10 seconds. Beyond spoken dialogue,
Voila is designed as a unified model for a wide range of voice-based
applications, including automatic speech recognition (ASR), Text-to-Speech
(TTS), and, with minimal adaptation, multilingual speech translation. Voila is
fully open-sourced to support open research and accelerate progress toward
next-generation human-machine interactions.
Voila is an end-to-end voice-language model that enables low-latency, emotionally expressive voice interactions and supports multiple applications including ASR, TTS, and multilingual speech translation.
end-to-end architecture full-duplex low-latency conversations hierarchical multi-scale Transformer acoustic modeling voice generation persona-aware automatic speech recognition (ASR) Text-to-Speech (TTS) multilingual speech translation