Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language
  Models
Model merging, such as model souping, is the practice of combining different
models with the same architecture together without further training. In this
work, we present a model merging methodology that addresses the difficulty of
fine-tuning Large Language Models (LLMs) for target tasks in non-English
languages, where task-specific data is often unavailable. We focus on
mathematical reasoning and without in-language math data, facilitate
cross-lingual transfer by composing language and math capabilities. Starting
from the same pretrained model, we fine-tune separate "experts" on math
instruction data in English and on generic instruction data in the target
language. We then replace the top and bottom transformer layers of the math
expert directly with layers from the language expert, which consequently
enhances math performance in the target language. The resulting merged models
outperform the individual experts and other merging methods on the math
benchmark, MGSM, by 10% across four major languages where math instruction data
is scarce. In addition, this layer swapping is simple, inexpensive, and
intuitive, as it is based on an interpretative analysis of the most important
parameter changes during the fine-tuning of each expert. The ability to
successfully re-compose LLMs for cross-lingual transfer in this manner opens up
future possibilities to combine model expertise, create modular solutions, and
transfer reasoning capabilities across languages all post hoc.
6700306bdb3b7a57f9494187
Lucas Bandarkar
claimed_verified
2024-10-10T08:02:11.013Z
6435b1d9aaef013d1aeea6a0
/avatars/1f515c67f3d03f35aa6e06f93566bd21.svg
Lucas Bandarkar
user
lucasbandarkar
6700306bdb3b7a57f9494188
Benjamin Muller
6700306bdb3b7a57f9494189
Pritish Yuvraj
claimed_verified
2024-10-04T20:11:17.248Z
65c30d162a023e7ea97c160f
https://cdn-avatars.huggingface.co/v1/production/uploads/65c30d162a023e7ea97c160f/WQ4ffvmAUODdKoVMri8N7.jpeg
Pritish Yuvraj
user
pyuvraj
6700306bdb3b7a57f949418a
Rui Hou
6700306bdb3b7a57f949418b
Nayan Singhal
6700306bdb3b7a57f949418c
Hongjiang Lv
6700306bdb3b7a57f949418d
Bing Liu
Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language
  Models
Model merging, such as model souping, is the practice of combining different
models with the same architecture together without further training. In this
work, we present a model merging methodology that addresses the difficulty of
fine-tuning Large Language Models (LLMs) for target tasks in non-English
languages, where task-specific data is often unavailable. We focus on
mathematical reasoning and without in-language math data, facilitate
cross-lingual transfer by composing language and math capabilities. Starting
from the same pretrained model, we fine-tune separate "experts" on math
instruction data in English and on generic instruction data in the target
language. We then replace the top and bottom transformer layers of the math
expert directly with layers from the language expert, which consequently
enhances math performance in the target language. The resulting merged models
outperform the individual experts and other merging methods on the math
benchmark, MGSM, by 10% across four major languages where math instruction data
is scarce. In addition, this layer swapping is simple, inexpensive, and
intuitive, as it is based on an interpretative analysis of the most important
parameter changes during the fine-tuning of each expert. The ability to
successfully re-compose LLMs for cross-lingual transfer in this manner opens up
future possibilities to combine model expertise, create modular solutions, and
transfer reasoning capabilities across languages all post hoc.
Merging pretrained models via layer swapping enhances mathematical reasoning in non-English languages where task-specific data is limited.
model souping Large Language Models (LLMs) mathematical reasoning cross-lingual transfer transformer layers fine-tuning math benchmark MGSM parameter changes model expertise modular solutions