Weighted-Reward Preference Optimization for Implicit Model Fusion
While fusing heterogeneous open-source LLMs with varying architectures and
sizes can potentially integrate the strengths of different models, existing
fusion methods face significant challenges, such as vocabulary alignment and
merging distribution matrices. These procedures are not only complex but also
prone to introducing noise and errors. In this paper, we propose an implicit
fusion method, Weighted-Reward Preference Optimization (WRPO), which leverages
preference optimization between the source LLMs and the target LLM to transfer
their capabilities effectively. WRPO eliminates the need for vocabulary
alignment and matrix fusion and can be efficiently scaled to accommodate
various LLMs. To address distributional deviations between the source and
target LLMs, WRPO introduces a progressive adaptation strategy that gradually
shifts reliance on preferred examples from the target LLM to the source LLMs.
Extensive experiments on the MT-Bench, AlpacaEval-2, and Arena-Hard benchmarks
demonstrate that WRPO consistently outperforms existing knowledge fusion
methods and various fine-tuning baselines. When applied to LLaMA3-8B-Instruct
as the target model, WRPO achieves a length-controlled win rate of 55.9%
against GPT-4-Preview-1106 on AlpacaEval-2 and a win rate of 46.2% against
GPT-4-0314 on Arena-Hard. Our code is available at
https://github.com/SLIT-AI/WRPO.
675146f4372c1020c21d2046
Ziyi Yang
claimed_verified
2024-12-06T10:37:58.424Z
64c9b0f28d2d187c24d1e6c1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/1CPnAaB3gsupdpiNWaoDc.png
ZiYi Yang
user
AALF
675146f4372c1020c21d2047
Fanqi Wan
admin_assigned
2024-12-05T12:23:54.907Z
62ecbffd99112e99c5f7fded
https://cdn-avatars.huggingface.co/v1/production/uploads/62ecbffd99112e99c5f7fded/U6iXAJbpm2vaC5qksEPiH.png
Fanqi Wan
user
Wanfq
675146f4372c1020c21d2048
Longguang Zhong
admin_assigned
2024-12-05T12:23:49.460Z
62b6d20416ff90e6198301b6
https://cdn-avatars.huggingface.co/v1/production/uploads/1656148456743-noauth.png
Longguang Zhong
user
GGLS
675146f4372c1020c21d2049
Tianyuan Shi
admin_assigned
2024-12-05T12:23:43.208Z
654c82ca47053d20c7986fe3
/avatars/b6560247e9d6f662d36aa1c1589bb92e.svg
tianyuan shi
user
stycoo
675146f4372c1020c21d204a
Xiaojun Quan
admin_assigned
2024-12-05T12:23:35.922Z
63b57d75bda8d44adf2ff3ff
/avatars/8a387036758b2f7fc7d7529dea206669.svg
Xiaojun Quan
user
passerqxj
Weighted-Reward Preference Optimization for Implicit Model Fusion
While fusing heterogeneous open-source LLMs with varying architectures and
sizes can potentially integrate the strengths of different models, existing
fusion methods face significant challenges, such as vocabulary alignment and
merging distribution matrices. These procedures are not only complex but also
prone to introducing noise and errors. In this paper, we propose an implicit
fusion method, Weighted-Reward Preference Optimization (WRPO), which leverages
preference optimization between the source LLMs and the target LLM to transfer
their capabilities effectively. WRPO eliminates the need for vocabulary
alignment and matrix fusion and can be efficiently scaled to accommodate
various LLMs. To address distributional deviations between the source and
target LLMs, WRPO introduces a progressive adaptation strategy that gradually
shifts reliance on preferred examples from the target LLM to the source LLMs.
Extensive experiments on the MT-Bench, AlpacaEval-2, and Arena-Hard benchmarks
demonstrate that WRPO consistently outperforms existing knowledge fusion
methods and various fine-tuning baselines. When applied to LLaMA3-8B-Instruct
as the target model, WRPO achieves a length-controlled win rate of 55.9%
against GPT-4-Preview-1106 on AlpacaEval-2 and a win rate of 46.2% against
GPT-4-0314 on Arena-Hard. Our code is available at
https://github.com/SLIT-AI/WRPO.
A novel implicit fusion method, WRPO, effectively transfers capabilities between heterogeneous LLMs, improving performance on benchmarks without manual vocabulary alignment or distribution matrix fusion.
heterogeneous open-source LLMs Weighted-Reward Preference Optimization (WRPO) preference optimization vocabulary alignment distribution matrices progressive adaptation strategy MT-Bench AlpacaEval-2 Arena-Hard LLaMA3-8B-Instruct GPT-4-Preview-1106 GPT-4-0314