Towards Achieving Human Parity on End-to-end Simultaneous Speech
  Translation via LLM Agent
In this paper, we present Cross Language Agent -- Simultaneous
Interpretation, CLASI, a high-quality and human-like Simultaneous Speech
Translation (SiST) System. Inspired by professional human interpreters, we
utilize a novel data-driven read-write strategy to balance the translation
quality and latency. To address the challenge of translating in-domain
terminologies, CLASI employs a multi-modal retrieving module to obtain relevant
information to augment the translation. Supported by LLMs, our approach can
generate error-tolerated translation by considering the input audio, historical
context, and retrieved information. Experimental results show that our system
outperforms other systems by significant margins. Aligned with professional
human interpreters, we evaluate CLASI with a better human evaluation metric,
valid information proportion (VIP), which measures the amount of information
that can be successfully conveyed to the listeners. In the real-world
scenarios, where the speeches are often disfluent, informal, and unclear, CLASI
achieves VIP of 81.3% and 78.0% for Chinese-to-English and English-to-Chinese
translation directions, respectively. In contrast, state-of-the-art commercial
or open-source systems only achieve 35.4% and 41.6%. On the extremely hard
dataset, where other systems achieve under 13% VIP, CLASI can still achieve 70%
VIP.
66aaf0de28277564bd72c56c
Shanbo Cheng
claimed_verified
2024-08-01T07:00:27.433Z
632ab0407fb39c2b6350c10a
/avatars/fb02cad2a017654965486418bf370157.svg
Cheng
user
Shanbo
66aaf0de28277564bd72c56d
Zhichao Huang
admin_assigned
2024-08-02T15:07:51.358Z
6411981aca1bb1e9253b3527
/avatars/f180b6def7636a9e6955ebcba744d05a.svg
Zhichao Huang
user
iceshzc
66aaf0de28277564bd72c56e
Tom Ko
66aaf0de28277564bd72c56f
Hang Li
66aaf0de28277564bd72c570
Ningxin Peng
admin_assigned
2024-08-02T15:08:33.028Z
64ae8b41c21d9baacbaa4d98
/avatars/82a09c5f390d7fb34b7092e03b793155.svg
Ningxin Peng
user
nxpeng9235
66aaf0de28277564bd72c571
Lu Xu
claimed_verified
2024-08-01T07:00:24.877Z
662c7c0a7b24fe598b317539
/avatars/9c578f8a1b9e5f85bdcfe0dc7465e0b4.svg
xuuuluuu
user
xuuuluuu
66aaf0de28277564bd72c572
Qini Zhang
Towards Achieving Human Parity on End-to-end Simultaneous Speech
  Translation via LLM Agent
In this paper, we present Cross Language Agent -- Simultaneous
Interpretation, CLASI, a high-quality and human-like Simultaneous Speech
Translation (SiST) System. Inspired by professional human interpreters, we
utilize a novel data-driven read-write strategy to balance the translation
quality and latency. To address the challenge of translating in-domain
terminologies, CLASI employs a multi-modal retrieving module to obtain relevant
information to augment the translation. Supported by LLMs, our approach can
generate error-tolerated translation by considering the input audio, historical
context, and retrieved information. Experimental results show that our system
outperforms other systems by significant margins. Aligned with professional
human interpreters, we evaluate CLASI with a better human evaluation metric,
valid information proportion (VIP), which measures the amount of information
that can be successfully conveyed to the listeners. In the real-world
scenarios, where the speeches are often disfluent, informal, and unclear, CLASI
achieves VIP of 81.3% and 78.0% for Chinese-to-English and English-to-Chinese
translation directions, respectively. In contrast, state-of-the-art commercial
or open-source systems only achieve 35.4% and 41.6%. On the extremely hard
dataset, where other systems achieve under 13% VIP, CLASI can still achieve 70%
VIP.
CLASI, a high-quality Simultaneous Speech Translation system, utilizes a data-driven strategy and multi-modal retrieving module to achieve superior translation quality compared to existing systems in various challenging scenarios.
Simultaneous Speech Translation (SiST) read-write strategy multi-modal retrieving module LLMs valid information proportion (VIP)