Tower+: Bridging Generality and Translation Specialization in
  Multilingual LLMs
Fine-tuning pretrained LLMs has been shown to be an effective strategy for
reaching state-of-the-art performance on specific tasks like machine
translation. However, this process of adaptation often implies sacrificing
general-purpose capabilities, such as conversational reasoning and
instruction-following, hampering the utility of the system in real-world
applications that require a mixture of skills. In this paper, we introduce
Tower+, a suite of models designed to deliver strong performance across both
translation and multilingual general-purpose text capabilities. We achieve a
Pareto frontier between translation specialization and multilingual
general-purpose capabilities by introducing a novel training recipe that builds
on Tower (Alves et al., 2024), comprising continued pretraining, supervised
fine-tuning, preference optimization, and reinforcement learning with
verifiable rewards. At each stage of training, we carefully generate and curate
data to strengthen performance on translation as well as general-purpose tasks
involving code generation, mathematics problem solving, and general
instruction-following. We develop models at multiple scales: 2B, 9B, and 72B.
Our smaller models often outperform larger general-purpose open-weight and
proprietary LLMs (e.g., Llama 3.3 70B, GPT-4o). Our largest model delivers
best-in-class translation performance for high-resource languages and top
results in multilingual Arena Hard evaluations and in IF-MT, a benchmark we
introduce for evaluating both translation and instruction-following. Our
findings highlight that it is possible to rival frontier models in general
capabilities, while optimizing for specific business domains, such as
translation and localization.
685bce34aafa002dfd2bdee0
Ricardo Rei
claimed_verified
2025-06-25T11:04:02.794Z
60d9a9791fa5d458da77754d
https://cdn-avatars.huggingface.co/v1/production/uploads/1624887555363-60d9a9791fa5d458da77754d.jpeg
Ricardo Rei
user
RicardoRei
685bce34aafa002dfd2bdee1
Nuno M. Guerreiro
claimed_verified
2025-07-22T08:33:34.502Z
61f06c8ba57920a251ec1a5c
https://cdn-avatars.huggingface.co/v1/production/uploads/61f06c8ba57920a251ec1a5c/TmmSN9xWm1noLeAtsUSRk.jpeg
Nuno M. Guerreiro
user
nunonmg
685bce34aafa002dfd2bdee2
José Pombal
claimed_verified
2025-07-02T07:41:24.810Z
6408b9d79e9f790c905398a2
/avatars/ee0661b6711073141eeb13dcc6b8b9bd.svg
José Maria Pombal
user
jmprcp
685bce34aafa002dfd2bdee3
João Alves
claimed_verified
2025-07-03T16:19:14.335Z
61ae32477b23f65fc71a62e4
/avatars/10054d6a17e65e5509f4f48cf4659cde.svg
João Alves
user
jalves97
685bce34aafa002dfd2bdee4
Pedro Teixeirinha
685bce34aafa002dfd2bdee5
Amin Farajian
685bce34aafa002dfd2bdee6
André F. T. Martins
Tower+: Bridging Generality and Translation Specialization in
  Multilingual LLMs
Fine-tuning pretrained LLMs has been shown to be an effective strategy for
reaching state-of-the-art performance on specific tasks like machine
translation. However, this process of adaptation often implies sacrificing
general-purpose capabilities, such as conversational reasoning and
instruction-following, hampering the utility of the system in real-world
applications that require a mixture of skills. In this paper, we introduce
Tower+, a suite of models designed to deliver strong performance across both
translation and multilingual general-purpose text capabilities. We achieve a
Pareto frontier between translation specialization and multilingual
general-purpose capabilities by introducing a novel training recipe that builds
on Tower (Alves et al., 2024), comprising continued pretraining, supervised
fine-tuning, preference optimization, and reinforcement learning with
verifiable rewards. At each stage of training, we carefully generate and curate
data to strengthen performance on translation as well as general-purpose tasks
involving code generation, mathematics problem solving, and general
instruction-following. We develop models at multiple scales: 2B, 9B, and 72B.
Our smaller models often outperform larger general-purpose open-weight and
proprietary LLMs (e.g., Llama 3.3 70B, GPT-4o). Our largest model delivers
best-in-class translation performance for high-resource languages and top
results in multilingual Arena Hard evaluations and in IF-MT, a benchmark we
introduce for evaluating both translation and instruction-following. Our
findings highlight that it is possible to rival frontier models in general
capabilities, while optimizing for specific business domains, such as
translation and localization.
Tower+, a suite of fine-tuned language models, achieves strong performance in both translation and multilingual general-purpose text tasks through a novel training recipe that includes continued pretraining, supervised fine-tuning, preference optimization, and reinforcement learning.
Tower+ continued pretraining supervised fine-tuning preference optimization reinforcement learning verifiable rewards multilingual general-purpose capabilities Tower LLaMA 3.3 70B GPT-4o multilingual Arena Hard IF-MT