Early-Exit and Instant Confidence Translation Quality Estimation
Quality estimation is omnipresent in machine translation, for both evaluation
and generation. Unfortunately, quality estimation models are often opaque and
computationally expensive, making them impractical to be part of large-scale
pipelines. In this work, we tackle two connected challenges: (1) reducing the
cost of quality estimation at scale, and (2) developing an inexpensive
uncertainty estimation method for quality estimation. To address the latter, we
introduce Instant Confidence COMET, an uncertainty-aware quality estimation
model that matches the performance of previous approaches at a fraction of
their costs. We extend this to Early-Exit COMET, a quality estimation model
that can compute quality scores and associated confidences already at early
model layers, allowing us to early-exit computations and reduce evaluation
costs. We also apply our model to machine translation reranking. We combine
Early-Exit COMET with an upper confidence bound bandit algorithm to find the
best candidate from a large pool without having to run the full evaluation
model on all candidates. In both cases (evaluation and reranking) our methods
reduce the required compute by 50% with very little degradation in performance.
67b835f98512a3eca052c0ee
Vilém Zouhar
claimed_verified
2025-02-25T14:09:46.337Z
6304ece07424d937fa35fb98
https://cdn-avatars.huggingface.co/v1/production/uploads/6304ece07424d937fa35fb98/6qZoqm-Ti8CiDcCHEt1sE.jpeg
Vilém Zouhar
user
zouharvi
67b835f98512a3eca052c0ef
Maike Züfle
admin_assigned
2025-02-25T16:38:17.623Z
6552004b4d9e71e17b35fa0b
/avatars/341ac1551588d5fcf5f3526fc06ff702.svg
Maike Züfle
user
maikez
67b835f98512a3eca052c0f0
Beni Egressy
admin_assigned
2025-02-25T16:38:24.223Z
6735d61d0c6b2cc068fe7cda
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/OD07myJrBbPaM1BUknLP-.png
Beni Egressy
user
egressbi
67b835f98512a3eca052c0f1
Julius Cheng
admin_assigned
2025-02-25T16:38:33.820Z
641af90d1911d3be6742928a
/avatars/d6d34c2d49cb8dfdccc6681aacf47cd4.svg
Julius Cheng
user
juliuscheng
67b835f98512a3eca052c0f2
Jan Niehues
admin_assigned
2025-02-25T16:38:40.110Z
63394998474cfeb1a85bde3f
/avatars/43d997eb118e6c6da9604e0c0bf0e63e.svg
Jan Niehues
user
jannieh
Early-Exit and Instant Confidence Translation Quality Estimation
Quality estimation is omnipresent in machine translation, for both evaluation
and generation. Unfortunately, quality estimation models are often opaque and
computationally expensive, making them impractical to be part of large-scale
pipelines. In this work, we tackle two connected challenges: (1) reducing the
cost of quality estimation at scale, and (2) developing an inexpensive
uncertainty estimation method for quality estimation. To address the latter, we
introduce Instant Confidence COMET, an uncertainty-aware quality estimation
model that matches the performance of previous approaches at a fraction of
their costs. We extend this to Early-Exit COMET, a quality estimation model
that can compute quality scores and associated confidences already at early
model layers, allowing us to early-exit computations and reduce evaluation
costs. We also apply our model to machine translation reranking. We combine
Early-Exit COMET with an upper confidence bound bandit algorithm to find the
best candidate from a large pool without having to run the full evaluation
model on all candidates. In both cases (evaluation and reranking) our methods
reduce the required compute by 50% with very little degradation in performance.
The paper presents lightweight quality and uncertainty estimation models for machine translation that significantly reduce computational costs while maintaining performance.
quality estimation machine translation uncertainty estimation instant confidence comet early-exit comet upper confidence bound bandit algorithm