Better Late Than Never: Evaluation of Latency Metrics for Simultaneous
  Speech-to-Text Translation
Simultaneous speech-to-text translation (SimulST) systems have to balance
translation quality with latency--the delay between speech input and the
translated output. While quality evaluation is well established, accurate
latency measurement remains a challenge. Existing metrics often produce
inconsistent or misleading results, especially in the widely used short-form
setting, where speech is artificially presegmented. In this paper, we present
the first comprehensive analysis of SimulST latency metrics across language
pairs, systems, and both short- and long-form regimes. We uncover a structural
bias in current metrics related to segmentation that undermines fair and
meaningful comparisons. To address this, we introduce YAAL (Yet Another Average
Lagging), a refined latency metric that delivers more accurate evaluations in
the short-form regime. We extend YAAL to LongYAAL for unsegmented audio and
propose SoftSegmenter, a novel resegmentation tool based on word-level
alignment. Our experiments show that YAAL and LongYAAL outperform popular
latency metrics, while SoftSegmenter enhances alignment quality in long-form
evaluation, together enabling more reliable assessments of SimulST systems.
68d2871f0e215259d193b041
Peter Polák
claimed_verified
2025-09-25T08:01:59.853Z
61b0b0521b3d95b3d1ed9256
/avatars/87701c5e32d59638d945699f38c0f9c8.svg
Peter Polak
user
peterpo
68d2871f0e215259d193b042
Sara Papi
claimed_verified
2025-09-24T13:54:25.376Z
66309b3833ccd9e68c5d5171
https://cdn-avatars.huggingface.co/v1/production/uploads/66309b3833ccd9e68c5d5171/UGt7pZazJHhtwjg0iiyBu.jpeg
Sara Papi
user
spapi
68d2871f0e215259d193b043
Luisa Bentivogli
admin_assigned
2025-09-24T15:31:19.179Z
662eb48f5ade870b60db4380
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ozHyP-GBIAEhR-l-TkBFb.jpeg
Luisa Bentivogli
user
lubentivogli
68d2871f0e215259d193b044
Ondřej Bojar
admin_assigned
2025-09-24T15:31:25.721Z
6551481e685ba4c13da2e244
/avatars/2455e45268b2b3cfd92bf7fa516ca229.svg
Ondrej Bojar
user
OndrejBojar
Better Late Than Never: Evaluation of Latency Metrics for Simultaneous
  Speech-to-Text Translation
Simultaneous speech-to-text translation (SimulST) systems have to balance
translation quality with latency--the delay between speech input and the
translated output. While quality evaluation is well established, accurate
latency measurement remains a challenge. Existing metrics often produce
inconsistent or misleading results, especially in the widely used short-form
setting, where speech is artificially presegmented. In this paper, we present
the first comprehensive analysis of SimulST latency metrics across language
pairs, systems, and both short- and long-form regimes. We uncover a structural
bias in current metrics related to segmentation that undermines fair and
meaningful comparisons. To address this, we introduce YAAL (Yet Another Average
Lagging), a refined latency metric that delivers more accurate evaluations in
the short-form regime. We extend YAAL to LongYAAL for unsegmented audio and
propose SoftSegmenter, a novel resegmentation tool based on word-level
alignment. Our experiments show that YAAL and LongYAAL outperform popular
latency metrics, while SoftSegmenter enhances alignment quality in long-form
evaluation, together enabling more reliable assessments of SimulST systems.
The paper analyzes SimulST latency metrics, identifies segmentation bias, and introduces YAAL and LongYAAL for more accurate latency evaluation, along with SoftSegmenter for improved alignment quality.
SimulST latency metrics short-form long-form segmentation bias YAAL LongYAAL SoftSegmenter word-level alignment