A Paradigm Shift in Machine Translation: Boosting Translation
  Performance of Large Language Models
Generative Large Language Models (LLMs) have achieved remarkable advancements
in various NLP tasks. However, these advances have not been reflected in the
translation task, especially those with moderate model sizes (i.e., 7B or 13B
parameters), which still lag behind conventional supervised encoder-decoder
translation models. Previous studies have attempted to improve the translation
capabilities of these moderate LLMs, but their gains have been limited. In this
study, we propose a novel fine-tuning approach for LLMs that is specifically
designed for the translation task, eliminating the need for the abundant
parallel data that traditional translation models usually depend on. Our
approach consists of two fine-tuning stages: initial fine-tuning on monolingual
data followed by subsequent fine-tuning on a small set of high-quality parallel
data. We introduce the LLM developed through this strategy as Advanced Language
Model-based trAnslator (ALMA). Based on LLaMA-2 as our underlying model, our
results show that the model can achieve an average improvement of more than 12
BLEU and 12 COMET over its zero-shot performance across 10 translation
directions from the WMT'21 (2 directions) and WMT'22 (8 directions) test
datasets. The performance is significantly better than all prior work and even
superior to the NLLB-54B model and GPT-3.5-text-davinci-003, with only 7B or
13B parameters. This method establishes the foundation for a novel training
paradigm in machine translation.
650ce69e23e8028a895c6259
Haoran Xu
admin_assigned
2023-09-22T15:15:51.545Z
61384b860317b0a5c10877d3
https://cdn-avatars.huggingface.co/v1/production/uploads/1631080954171-61384b860317b0a5c10877d3.jpeg
Haoran Xu
user
haoranxu
650ce69e23e8028a895c625a
Young Jin Kim
admin_assigned
2023-09-22T15:16:40.868Z
63f5173bb51da4d61da6c038
/avatars/0ee530cf80476aa3985c4d591cd384a1.svg
Young Jin Kim
user
ykim362
650ce69e23e8028a895c625b
Amr Sharaf
admin_assigned
2023-09-22T15:16:49.141Z
619ffce1ea7f7717b42344c6
/avatars/95bbb992cadfc8c3523ec2850d6e1e65.svg
Amr Sharaf
user
amrsharaf
650ce69e23e8028a895c625c
Hany Hassan Awadalla
claimed_verified
2023-09-22T17:53:43.613Z
6290f299d1be2630c0d30e1c
/avatars/6aff70159d1930732d5ffeed54761351.svg
Hany
user
hanyh
A Paradigm Shift in Machine Translation: Boosting Translation
  Performance of Large Language Models
Generative Large Language Models (LLMs) have achieved remarkable advancements
in various NLP tasks. However, these advances have not been reflected in the
translation task, especially those with moderate model sizes (i.e., 7B or 13B
parameters), which still lag behind conventional supervised encoder-decoder
translation models. Previous studies have attempted to improve the translation
capabilities of these moderate LLMs, but their gains have been limited. In this
study, we propose a novel fine-tuning approach for LLMs that is specifically
designed for the translation task, eliminating the need for the abundant
parallel data that traditional translation models usually depend on. Our
approach consists of two fine-tuning stages: initial fine-tuning on monolingual
data followed by subsequent fine-tuning on a small set of high-quality parallel
data. We introduce the LLM developed through this strategy as Advanced Language
Model-based trAnslator (ALMA). Based on LLaMA-2 as our underlying model, our
results show that the model can achieve an average improvement of more than 12
BLEU and 12 COMET over its zero-shot performance across 10 translation
directions from the WMT'21 (2 directions) and WMT'22 (8 directions) test
datasets. The performance is significantly better than all prior work and even
superior to the NLLB-54B model and GPT-3.5-text-davinci-003, with only 7B or
13B parameters. This method establishes the foundation for a novel training
paradigm in machine translation.
A novel fine-tuning approach for moderate-sized language models significantly improves translation performance with less parallel data compared to traditional methods.
Generative Large Language Models fine-tuning monolingual data parallel data Advanced Language Model-based trAnslator (ALMA) zero-shot performance BLEU COMET WMT'21 WMT'22 NLLB-54B GPT-3.5-text-davinci-003