Model Merging in Pre-training of Large Language Models
Model merging has emerged as a promising technique for enhancing large
language models, though its application in large-scale pre-training remains
relatively unexplored. In this paper, we present a comprehensive investigation
of model merging techniques during the pre-training process. Through extensive
experiments with both dense and Mixture-of-Experts (MoE) architectures ranging
from millions to over 100 billion parameters, we demonstrate that merging
checkpoints trained with constant learning rates not only achieves significant
performance improvements but also enables accurate prediction of annealing
behavior. These improvements lead to both more efficient model development and
significantly lower training costs. Our detailed ablation studies on merging
strategies and hyperparameters provide new insights into the underlying
mechanisms while uncovering novel applications. Through comprehensive
experimental analysis, we offer the open-source community practical
pre-training guidelines for effective model merging.
682c55ad39dfc753340da444
Yunshui Li
claimed_verified
2025-06-19T15:11:35.246Z
62e670d33651180f7d334ef3
https://cdn-avatars.huggingface.co/v1/production/uploads/1661829952739-62e670d33651180f7d334ef3.jpeg
Li Yunshui
user
Wa2erGo
682c55ad39dfc753340da445
Yiyuan Ma
682c55ad39dfc753340da446
Shen Yan
682c55ad39dfc753340da447
Chaoyi Zhang
682c55ad39dfc753340da448
Jing Liu
682c55ad39dfc753340da449
Jianqiao Lu
682c55ad39dfc753340da44a
Ziwen Xu
682c55ad39dfc753340da44b
Mengzhao Chen
claimed_verified
2025-05-22T07:18:09.440Z
64aea082704210bf815e7551
/avatars/5c8dc0df57596c526b2bccea21835f53.svg
Mengzhao Chen
user
ChenMnZ
682c55ad39dfc753340da44c
Minrui Wang
682c55ad39dfc753340da44d
Shiyi Zhan
682c55ad39dfc753340da44e
Jin Ma
682c55ad39dfc753340da44f
Xunhao Lai
682c55ad39dfc753340da450
Yao Luo
682c55ad39dfc753340da451
Xingyan Bin
682c55ad39dfc753340da452
Hongbin Ren
682c55ad39dfc753340da453
Mingji Han
682c55ad39dfc753340da454
Wenhao Hao
682c55ad39dfc753340da455
Bairen Yi
claimed_verified
2025-05-21T09:03:30.697Z
6369d92f64aad59d4d44d362
/avatars/73956400cfbfd53116aefc17b3c9f0fd.svg
Yi
user
Bairen
682c55ad39dfc753340da456
LingJun Liu
682c55ad39dfc753340da457
Bole Ma
682c55ad39dfc753340da458
Xiaoying Jia
682c55ad39dfc753340da459
Zhou Xun
682c55ad39dfc753340da45a
Liang Xiang
682c55ad39dfc753340da45b
Yonghui Wu
Model Merging in Pre-training of Large Language Models
Model merging has emerged as a promising technique for enhancing large
language models, though its application in large-scale pre-training remains
relatively unexplored. In this paper, we present a comprehensive investigation
of model merging techniques during the pre-training process. Through extensive
experiments with both dense and Mixture-of-Experts (MoE) architectures ranging
from millions to over 100 billion parameters, we demonstrate that merging
checkpoints trained with constant learning rates not only achieves significant
performance improvements but also enables accurate prediction of annealing
behavior. These improvements lead to both more efficient model development and
significantly lower training costs. Our detailed ablation studies on merging
strategies and hyperparameters provide new insights into the underlying
mechanisms while uncovering novel applications. Through comprehensive
experimental analysis, we offer the open-source community practical
pre-training guidelines for effective model merging.
Model merging during pre-training enhances large language models, improving performance and reducing costs across various architectures and learning rates.
dense architectures Mixture-of-Experts checkpoints constant learning rates annealing behavior ablation studies hyperparameters pre-training guidelines