What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine
  Translation with a Human-centered Study
Gender bias in machine translation (MT) is recognized as an issue that can
harm people and society. And yet, advancements in the field rarely involve
people, the final MT users, or inform how they might be impacted by biased
technologies. Current evaluations are often restricted to automatic methods,
which offer an opaque estimate of what the downstream impact of gender
disparities might be. We conduct an extensive human-centered study to examine
if and to what extent bias in MT brings harms with tangible costs, such as
quality of service gaps across women and men. To this aim, we collect
behavioral data from 90 participants, who post-edited MT outputs to ensure
correct gender translation. Across multiple datasets, languages, and types of
users, our study shows that feminine post-editing demands significantly more
technical and temporal effort, also corresponding to higher financial costs.
Existing bias measurements, however, fail to reflect the found disparities. Our
findings advocate for human-centered approaches that can inform the societal
impact of bias.
66fd206f9bd05d6ff6910ba3
Beatrice Savoldi
admin_assigned
2024-10-03T09:36:43.306Z
63b58c53bda8d44adf3092de
https://cdn-avatars.huggingface.co/v1/production/uploads/63b58c53bda8d44adf3092de/7Qs7MmPHmQtZ7Hno5n6-P.jpeg
Beatrice Savoldi
user
BSavoldi
66fd206f9bd05d6ff6910ba4
Sara Papi
admin_assigned
2024-10-03T09:37:43.497Z
66309b3833ccd9e68c5d5171
https://cdn-avatars.huggingface.co/v1/production/uploads/66309b3833ccd9e68c5d5171/UGt7pZazJHhtwjg0iiyBu.jpeg
Sara Papi
user
spapi
66fd206f9bd05d6ff6910ba5
Matteo Negri
admin_assigned
2024-10-03T09:37:04.221Z
66350219843f549fdac86347
/avatars/1ad7aa4e8e6d80c5d50bf4de502f11a4.svg
Matteo Negri
user
MNegri
66fd206f9bd05d6ff6910ba6
Ana Guerberof
66fd206f9bd05d6ff6910ba7
Luisa Bentivogli
admin_assigned
2024-10-03T09:37:21.028Z
662eb48f5ade870b60db4380
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ozHyP-GBIAEhR-l-TkBFb.jpeg
Luisa Bentivogli
user
lubentivogli
What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine
  Translation with a Human-centered Study
Gender bias in machine translation (MT) is recognized as an issue that can
harm people and society. And yet, advancements in the field rarely involve
people, the final MT users, or inform how they might be impacted by biased
technologies. Current evaluations are often restricted to automatic methods,
which offer an opaque estimate of what the downstream impact of gender
disparities might be. We conduct an extensive human-centered study to examine
if and to what extent bias in MT brings harms with tangible costs, such as
quality of service gaps across women and men. To this aim, we collect
behavioral data from 90 participants, who post-edited MT outputs to ensure
correct gender translation. Across multiple datasets, languages, and types of
users, our study shows that feminine post-editing demands significantly more
technical and temporal effort, also corresponding to higher financial costs.
Existing bias measurements, however, fail to reflect the found disparities. Our
findings advocate for human-centered approaches that can inform the societal
impact of bias.
A human-centered study reveals that gender bias in machine translation imposes higher costs and effort on feminine post-editing, highlighting the need for more inclusive evaluation methods.
machine translation gender bias post-editing societal impact gender disparities