Xmodel-2 Technical Report
Xmodel-2 is a 1.2-billion-parameter large language model designed
specifically for reasoning tasks. Its architecture enables different model
scales to share a unified set of hyperparameters, allowing for extensive
experimentation on smaller models and seamless transfer of optimal
configurations to larger models. To maximize training efficiency and stability,
Xmodel-2 employs the WSD learning rate scheduler from MiniCPM. Pretrained on
1.5 trillion tokens from diverse sources, Xmodel-2 achieves state-of-the-art
performance in complex reasoning and agent-based tasks, while maintaining low
training costs. These results highlight the potential of efficient model design
and training strategies in advancing reasoning capabilities. Model checkpoints
and code are publicly available on GitHub at
https://github.com/XiaoduoAILab/Xmodel-2
677223b635722632fcc63ff5
Wang Qun
claimed_verified
2024-12-30T19:29:25.463Z
647adfd0e3a7d24c8e46d7d1
https://cdn-avatars.huggingface.co/v1/production/uploads/647adfd0e3a7d24c8e46d7d1/UqZegkj3P2XyPe4Lgnrt7.jpeg
ValeriaWong
user
valeriaWong
677223b635722632fcc63ff6
Liu Yang
677223b635722632fcc63ff7
Lin Qingquan
677223b635722632fcc63ff8
Qu Zhijiu
677223b635722632fcc63ff9
Jiang Ling
Xmodel-2 Technical Report
Xmodel-2 is a 1.2-billion-parameter large language model designed
specifically for reasoning tasks. Its architecture enables different model
scales to share a unified set of hyperparameters, allowing for extensive
experimentation on smaller models and seamless transfer of optimal
configurations to larger models. To maximize training efficiency and stability,
Xmodel-2 employs the WSD learning rate scheduler from MiniCPM. Pretrained on
1.5 trillion tokens from diverse sources, Xmodel-2 achieves state-of-the-art
performance in complex reasoning and agent-based tasks, while maintaining low
training costs. These results highlight the potential of efficient model design
and training strategies in advancing reasoning capabilities. Model checkpoints
and code are publicly available on GitHub at
https://github.com/XiaoduoAILab/Xmodel-2
Xmodel-2, a 1.2-billion-parameter large language model, utilizes a unified set of hyperparameters across different scales and the WSD learning rate scheduler to achieve state-of-the-art performance in complex reasoning tasks with efficient training.
large language model reasoning tasks unified set of hyperparameters WSD learning rate scheduler MiniCPM pretrained agent-based tasks