Spatial Speech Translation: Translating Across Space With Binaural
  Hearables
Imagine being in a crowded space where people speak a different language and
having hearables that transform the auditory space into your native language,
while preserving the spatial cues for all speakers. We introduce spatial speech
translation, a novel concept for hearables that translate speakers in the
wearer's environment, while maintaining the direction and unique voice
characteristics of each speaker in the binaural output. To achieve this, we
tackle several technical challenges spanning blind source separation,
localization, real-time expressive translation, and binaural rendering to
preserve the speaker directions in the translated audio, while achieving
real-time inference on the Apple M2 silicon. Our proof-of-concept evaluation
with a prototype binaural headset shows that, unlike existing models, which
fail in the presence of interference, we achieve a BLEU score of up to 22.01
when translating between languages, despite strong interference from other
speakers in the environment. User studies further confirm the system's
effectiveness in spatially rendering the translated speech in previously unseen
real-world reverberant environments. Taking a step back, this work marks the
first step towards integrating spatial perception into speech translation.
68151b71df379a1f7e1ce0ef
Tuochao Chen
claimed_verified
2025-05-04T10:04:34.322Z
65a9d16f3b9e1f0f307c96c3
/avatars/882a0746a0adb2d7befe563f7fbc6321.svg
Tuochao Chen
user
tuochao
68151b71df379a1f7e1ce0f0
Qirui Wang
68151b71df379a1f7e1ce0f1
Runlin He
68151b71df379a1f7e1ce0f2
Shyam Gollakota
Spatial Speech Translation: Translating Across Space With Binaural
  Hearables
Imagine being in a crowded space where people speak a different language and
having hearables that transform the auditory space into your native language,
while preserving the spatial cues for all speakers. We introduce spatial speech
translation, a novel concept for hearables that translate speakers in the
wearer's environment, while maintaining the direction and unique voice
characteristics of each speaker in the binaural output. To achieve this, we
tackle several technical challenges spanning blind source separation,
localization, real-time expressive translation, and binaural rendering to
preserve the speaker directions in the translated audio, while achieving
real-time inference on the Apple M2 silicon. Our proof-of-concept evaluation
with a prototype binaural headset shows that, unlike existing models, which
fail in the presence of interference, we achieve a BLEU score of up to 22.01
when translating between languages, despite strong interference from other
speakers in the environment. User studies further confirm the system's
effectiveness in spatially rendering the translated speech in previously unseen
real-world reverberant environments. Taking a step back, this work marks the
first step towards integrating spatial perception into speech translation.
A binaural spatial speech translation system achieves high translation accuracy while preserving speaker directional cues, even in noisy environments.
spatial speech translation blind source separation localization real-time expressive translation binaural rendering real-time inference BLEU score binaural headset spatial perception