HARE: HumAn pRiors, a key to small language model Efficiency
Human priors play a crucial role in efficiently utilizing data in deep
learning. However, with the development of large language models (LLMs), there
is an increasing emphasis on scaling both model size and data volume, which
often diminishes the importance of human priors in data construction.
Influenced by these trends, existing Small Language Models (SLMs) mainly rely
on web-scraped large-scale training data, neglecting the proper incorporation
of human priors. This oversight limits the training efficiency of language
models in resource-constrained settings. In this paper, we propose a principle
to leverage human priors for data construction. This principle emphasizes
achieving high-performance SLMs by training on a concise dataset that
accommodates both semantic diversity and data quality consistency, while
avoiding benchmark data leakage. Following this principle, we train an SLM
named HARE-1.1B. Extensive experiments on large-scale benchmark datasets
demonstrate that HARE-1.1B performs favorably against state-of-the-art SLMs,
validating the effectiveness of the proposed principle. Additionally, this
provides new insights into efficient language model training in
resource-constrained environments from the view of human priors.
66717c101f4de3ba8b1dfbb1
Lingyun Zhang
claimed_verified
2024-06-21T07:07:05.113Z
64eef8cfa331d9798cb18015
/avatars/4c1b1e0628d2d0f321ba5fe1a0ec8c03.svg
Zhang
user
lingyun1
66717c101f4de3ba8b1dfbb2
Bin jin
66717c101f4de3ba8b1dfbb3
Gaojian Ge
admin_assigned
2024-06-21T11:22:57.410Z
6662e38970c1a3233ca611d3
/avatars/60a8a84199cb67b4276cd83bbe1991a6.svg
gegaojian
user
fo1ge
66717c101f4de3ba8b1dfbb4
Lunhui Liu
admin_assigned
2024-06-21T11:23:04.909Z
653218e5ef70d3555f4831f8
/avatars/632f1cd0adededc1d495b9eb55d8adc2.svg
LunHui Liu
user
LLHLLH
66717c101f4de3ba8b1dfbb5
Xuewen Shen
66717c101f4de3ba8b1dfbb6
Mingyong Wu
admin_assigned
2024-06-21T11:23:24.373Z
65eec2e0c9ef20fb409a15b9
/avatars/a5a7c688de2f5f4fdef6b9ae91b98043.svg
wumingyong
user
wumingyong
66717c101f4de3ba8b1dfbb7
Houqian Zhang
66717c101f4de3ba8b1dfbb8
Yongneng Jiang
66717c101f4de3ba8b1dfbb9
Shiqi Chen
admin_assigned
2024-06-21T11:23:48.808Z
62bf99cae140faaa85335ab8
/avatars/bcd2aa1823995b385096e2a68ce3e071.svg
Shiqi Chen
user
ShiqiChen
66717c101f4de3ba8b1dfbba
Shi Pu
admin_assigned
2024-06-21T11:24:19.644Z
63968489fa66eacbf4a34812
/avatars/8d3a9f053e99597c82d8d5ab491d913f.svg
Shi Pu
user
pushi
HARE: HumAn pRiors, a key to small language model Efficiency
Human priors play a crucial role in efficiently utilizing data in deep
learning. However, with the development of large language models (LLMs), there
is an increasing emphasis on scaling both model size and data volume, which
often diminishes the importance of human priors in data construction.
Influenced by these trends, existing Small Language Models (SLMs) mainly rely
on web-scraped large-scale training data, neglecting the proper incorporation
of human priors. This oversight limits the training efficiency of language
models in resource-constrained settings. In this paper, we propose a principle
to leverage human priors for data construction. This principle emphasizes
achieving high-performance SLMs by training on a concise dataset that
accommodates both semantic diversity and data quality consistency, while
avoiding benchmark data leakage. Following this principle, we train an SLM
named HARE-1.1B. Extensive experiments on large-scale benchmark datasets
demonstrate that HARE-1.1B performs favorably against state-of-the-art SLMs,
validating the effectiveness of the proposed principle. Additionally, this
provides new insights into efficient language model training in
resource-constrained environments from the view of human priors.
A principle for leveraging human priors in data construction is proposed to improve small language models, demonstrating favorable performance on large benchmarks in resource-constrained settings.
large language models small language models human priors data construction semantic diversity data quality consistency benchmark data leakage HARE-1.1B