Rho-1: Not All Tokens Are What You Need
Previous language model pre-training methods have uniformly applied a
next-token prediction loss to all training tokens. Challenging this norm, we
posit that "Not all tokens in a corpus are equally important for language model
training". Our initial analysis delves into token-level training dynamics of
language model, revealing distinct loss patterns for different tokens.
Leveraging these insights, we introduce a new language model called Rho-1.
Unlike traditional LMs that learn to predict every next token in a corpus,
Rho-1 employs Selective Language Modeling (SLM), which selectively trains on
useful tokens that aligned with the desired distribution. This approach
involves scoring pretraining tokens using a reference model, and then training
the language model with a focused loss on tokens with higher excess loss. When
continual pretraining on 15B OpenWebMath corpus, Rho-1 yields an absolute
improvement in few-shot accuracy of up to 30% in 9 math tasks. After
fine-tuning, Rho-1-1B and 7B achieved state-of-the-art results of 40.6% and
51.8% on MATH dataset, respectively - matching DeepSeekMath with only 3% of the
pretraining tokens. Furthermore, when pretraining on 80B general tokens, Rho-1
achieves 6.8% average enhancement across 15 diverse tasks, increasing both
efficiency and performance of the language model pre-training.
6618914a507806c4e6796333
Zhenghao Lin
claimed_verified
2024-04-16T00:02:49.326Z
63776f1806241efce1e7aae6
/avatars/d67d9dcd932934c630f407ac152f2ce6.svg
Zhenghao Lin
user
Lin0
6618914a507806c4e6796334
Zhibin Gou
claimed_verified
2024-04-12T04:19:48.766Z
6521678b7bf8cc2dd2a4b02a
/avatars/3a0aa2f517d8709059a9ecbe83ddfc8a.svg
Zhibin Gou (alt)
user
zhibin-msft
6618914a507806c4e6796335
Yeyun Gong
admin_assigned
2024-04-18T11:50:18.623Z
643f615aa16cd6d1f4c581de
/avatars/47753a3e82b44f81881600c52e1e8495.svg
Yeyun Gong
user
yegong
6618914a507806c4e6796336
Xiao Liu
claimed_verified
2024-04-15T06:46:08.307Z
63fb6e281b4b1bd4e7ffc5be
https://cdn-avatars.huggingface.co/v1/production/uploads/63fb6e281b4b1bd4e7ffc5be/aiRu_bulgnxvEMrjipGoQ.jpeg
Xiao Liu
user
lx865712528
6618914a507806c4e6796337
Yelong Shen
admin_assigned
2024-04-18T11:50:25.406Z
6454c337a13edf669cd5d8ea
/avatars/a383a0dda7c2ef6a0d6c3c64651f42ff.svg
Yelong Shen
user
uuu6
6618914a507806c4e6796338
Ruochen Xu
admin_assigned
2024-04-18T11:50:39.473Z
64a234a8d94c3f9ccec8476d
/avatars/633adf359b1a65f458c3b5a7e59f11c0.svg
Ruochen Xu
user
ruochenx
6618914a507806c4e6796339
Chen Lin
6618914a507806c4e679633a
Yujiu Yang
admin_assigned
2024-04-18T11:50:56.018Z
64ca1fe838837b12d5e529b7
/avatars/44a3ad9e59318784ac531993b5f69f6b.svg
Yujiu Yang
user
Thu-redrobot
6618914a507806c4e679633b
Jian Jiao
6618914a507806c4e679633c
Nan Duan
6618914a507806c4e679633d
Weizhu Chen
admin_assigned
2024-04-18T11:51:22.643Z
64da876370446182be5b608d
/avatars/e412fdc71404ecdf638e416846e3ebfb.svg
Weizhu Chen
user
chenweizhu
Rho-1: Not All Tokens Are What You Need
Previous language model pre-training methods have uniformly applied a
next-token prediction loss to all training tokens. Challenging this norm, we
posit that "Not all tokens in a corpus are equally important for language model
training". Our initial analysis delves into token-level training dynamics of
language model, revealing distinct loss patterns for different tokens.
Leveraging these insights, we introduce a new language model called Rho-1.
Unlike traditional LMs that learn to predict every next token in a corpus,
Rho-1 employs Selective Language Modeling (SLM), which selectively trains on
useful tokens that aligned with the desired distribution. This approach
involves scoring pretraining tokens using a reference model, and then training
the language model with a focused loss on tokens with higher excess loss. When
continual pretraining on 15B OpenWebMath corpus, Rho-1 yields an absolute
improvement in few-shot accuracy of up to 30% in 9 math tasks. After
fine-tuning, Rho-1-1B and 7B achieved state-of-the-art results of 40.6% and
51.8% on MATH dataset, respectively - matching DeepSeekMath with only 3% of the
pretraining tokens. Furthermore, when pretraining on 80B general tokens, Rho-1
achieves 6.8% average enhancement across 15 diverse tasks, increasing both
efficiency and performance of the language model pre-training.
Rho-1, a novel language model using Selective Language Modeling, improves efficiency and performance by selectively training on useful tokens rather than all tokens in the corpus.
next-token prediction token-level training dynamics Selective Language Modeling SLM reference model excess loss few-shot accuracy MATH dataset DeepSeekMath pretraining tokens efficiency performance