Ziya2: Data-centric Learning is All LLMs Need
Various large language models (LLMs) have been proposed in recent years,
including closed- and open-source ones, continually setting new records on
multiple benchmarks. However, the development of LLMs still faces several
issues, such as high cost of training models from scratch, and continual
pre-training leading to catastrophic forgetting, etc. Although many such issues
are addressed along the line of research on LLMs, an important yet practical
limitation is that many studies overly pursue enlarging model sizes without
comprehensively analyzing and optimizing the use of pre-training data in their
learning process, as well as appropriate organization and leveraging of such
data in training LLMs under cost-effective settings. In this work, we propose
Ziya2, a model with 13 billion parameters adopting LLaMA2 as the foundation
model, and further pre-trained on 700 billion tokens, where we focus on
pre-training techniques and use data-centric optimization to enhance the
learning process of Ziya2 on different stages. Experiments show that Ziya2
significantly outperforms other models in multiple benchmarks especially with
promising results compared to representative open-source ones. Ziya2 (Base) is
released at https://huggingface.co/IDEA-CCNL/Ziya2-13B-Base and
https://modelscope.cn/models/Fengshenbang/Ziya2-13B-Base/summary.
6549b80f44e75a7de4fb301d
Ruyi Gan
admin_assigned
2023-11-07T15:41:15.198Z
612300897d70fa0063f38731
https://cdn-avatars.huggingface.co/v1/production/uploads/1637806723246-612300897d70fa0063f38731.jpeg
ganruyi
user
roygan
6549b80f44e75a7de4fb301e
Ziwei Wu
admin_assigned
2023-11-07T15:41:28.329Z
619f0825678725e00cd9a27f
https://cdn-avatars.huggingface.co/v1/production/uploads/1637812566433-619f0825678725e00cd9a27f.png
wuziwei
user
Zimix
6549b80f44e75a7de4fb301f
Renliang Sun
admin_assigned
2023-11-07T15:41:36.227Z
653a195b0da86d726c9c580c
/avatars/61649e1d600fdc1edc50ead0dfa99fdd.svg
Renliang Sun
user
RLSNLP
6549b80f44e75a7de4fb3020
Junyu Lu
6549b80f44e75a7de4fb3021
Xiaojun Wu
claimed_verified
2024-01-29T11:43:42.071Z
6352637d0f9bdb641c44e52d
https://cdn-avatars.huggingface.co/v1/production/uploads/6352637d0f9bdb641c44e52d/mSBRPzcH5pIV68PUmcsHV.png
wuxiaojun
user
wuxiaojun
6549b80f44e75a7de4fb3022
Dixiang Zhang
6549b80f44e75a7de4fb3023
Kunhao Pan
admin_assigned
2023-11-07T15:42:19.278Z
635262337d071f23d084d8af
https://cdn-avatars.huggingface.co/v1/production/uploads/1666695219265-635262337d071f23d084d8af.jpeg
Kunhao Pan
user
pskun
6549b80f44e75a7de4fb3024
Ping Yang
6549b80f44e75a7de4fb3025
Qi Yang
admin_assigned
2023-11-07T15:42:50.708Z
6215a6efbfcb3893344dd095
https://cdn-avatars.huggingface.co/v1/production/uploads/1668779559428-6215a6efbfcb3893344dd095.png
Qi Yang
user
qiyang
6549b80f44e75a7de4fb3026
Jiaxing Zhang
admin_assigned
2023-11-07T15:42:58.477Z
637854b38f288aba3d01d261
/avatars/a7f0daf90326bdfea134c75b0bc9bea1.svg
Jiaxing Zhang
user
ideazhangjason
6549b80f44e75a7de4fb3027
Yan Song
Ziya2: Data-centric Learning is All LLMs Need
Various large language models (LLMs) have been proposed in recent years,
including closed- and open-source ones, continually setting new records on
multiple benchmarks. However, the development of LLMs still faces several
issues, such as high cost of training models from scratch, and continual
pre-training leading to catastrophic forgetting, etc. Although many such issues
are addressed along the line of research on LLMs, an important yet practical
limitation is that many studies overly pursue enlarging model sizes without
comprehensively analyzing and optimizing the use of pre-training data in their
learning process, as well as appropriate organization and leveraging of such
data in training LLMs under cost-effective settings. In this work, we propose
Ziya2, a model with 13 billion parameters adopting LLaMA2 as the foundation
model, and further pre-trained on 700 billion tokens, where we focus on
pre-training techniques and use data-centric optimization to enhance the
learning process of Ziya2 on different stages. Experiments show that Ziya2
significantly outperforms other models in multiple benchmarks especially with
promising results compared to representative open-source ones. Ziya2 (Base) is
released at https://huggingface.co/IDEA-CCNL/Ziya2-13B-Base and
https://modelscope.cn/models/Fengshenbang/Ziya2-13B-Base/summary.
Ziya2, a 13-billion-parameter language model built on LLaMA2 and further pre-trained on 700 billion tokens, achieves superior performance across multiple benchmarks using data-centric optimization techniques.
large language models LLMs closed-source open-source catastrophic forgetting pre-training data cost-effective settings data-centric optimization Ziya2 LLaMA2