TÜLU 3: Pushing Frontiers in Open Language Model Post-Training
Language model post-training is applied to refine behaviors and unlock new
skills across a wide range of recent language models, but open recipes for
applying these techniques lag behind proprietary ones. The underlying training
data and recipes for post-training are simultaneously the most important pieces
of the puzzle and the portion with the least transparency. To bridge this gap,
we introduce T\"ULU 3, a family of fully-open state-of-the-art post-trained
models, alongside its data, code, and training recipes, serving as a
comprehensive guide for modern post-training techniques. T\"ULU 3, which builds
on Llama 3.1 base models, achieves results surpassing the instruct versions of
Llama 3.1, Qwen 2.5, Mistral, and even closed models such as GPT-4o-mini and
Claude 3.5-Haiku. The training algorithms for our models include supervised
finetuning (SFT), Direct Preference Optimization (DPO), and a novel method we
call Reinforcement Learning with Verifiable Rewards (RLVR). With T\"ULU 3, we
introduce a multi-task evaluation scheme for post-training recipes with
development and unseen evaluations, standard benchmark implementations, and
substantial decontamination of existing open datasets on said benchmarks. We
conclude with analysis and discussion of training methods that did not reliably
improve performance.
  In addition to the T\"ULU 3 model weights and demo, we release the complete
recipe -- including datasets for diverse core skills, a robust toolkit for data
curation and evaluation, the training code and infrastructure, and, most
importantly, a detailed report for reproducing and further adapting the T\"ULU
3 approach to more domains.
674414c8d3ad4510c1a26134
Nathan Lambert
admin_assigned
2024-11-25T20:52:52.869Z
628e5f90a9a3c754c1f7c88f
https://cdn-avatars.huggingface.co/v1/production/uploads/628e5f90a9a3c754c1f7c88f/iWqMY_l6dalrgRaJZWbK3.png
Nathan Lambert
user
natolambert
674414c8d3ad4510c1a26135
Jacob Morrison
admin_assigned
2024-11-25T21:40:52.602Z
63f51f6cb51da4d61da72937
/avatars/6023ecaafb28b7599f59816381d58bd0.svg
Jacob Morrison
user
jacobmorrison
674414c8d3ad4510c1a26136
Valentina Pyatkin
admin_assigned
2024-11-25T21:40:58.240Z
6556cff80e7a7067a934445f
https://cdn-avatars.huggingface.co/v1/production/uploads/6556cff80e7a7067a934445f/PoT7qQ6tVqLGGrYvGBbr8.jpeg
Valentina Pyatkin
user
valpy
674414c8d3ad4510c1a26137
Shengyi Huang
claimed_verified
2025-01-05T23:03:13.686Z
62bfc80b54db40ca617ce53a
https://cdn-avatars.huggingface.co/v1/production/uploads/62bfc80b54db40ca617ce53a/IZxafMqoYJNHwTuFt8AqF.png
Shengyi Costa Huang
user
vwxyzjn
674414c8d3ad4510c1a26138
Hamish Ivison
admin_assigned
2024-11-25T21:41:19.991Z
62608fc2ffe8827cb1d89f9f
https://cdn-avatars.huggingface.co/v1/production/uploads/1654027835241-62608fc2ffe8827cb1d89f9f.png
Hamish Ivison
user
hamishivi
674414c8d3ad4510c1a26139
Faeze Brahman
admin_assigned
2024-11-25T21:41:25.755Z
65282b8d578679aac7888aec
https://cdn-avatars.huggingface.co/v1/production/uploads/65282b8d578679aac7888aec/dibBkhH-z1c70mJZZxJ7u.jpeg
Faeze Brahman
user
faezeb
674414c8d3ad4510c1a2613a
Lester James V. Miranda
claimed_verified
2024-11-26T21:21:20.277Z
634e20a0c1ce28f1de920cc4
https://cdn-avatars.huggingface.co/v1/production/uploads/1666064515342-noauth.jpeg
Lj V. Miranda
user
ljvmiranda921
674414c8d3ad4510c1a2613b
Alisa Liu
674414c8d3ad4510c1a2613c
Nouha Dziri
admin_assigned
2024-11-25T21:41:51.152Z
625888c2c964680d85d70bff
/avatars/2ba91e452c5ea577ec9f1f71abc2fa49.svg
Nouha Dziri
user
nouhadziri
674414c8d3ad4510c1a2613d
Shane Lyu
674414c8d3ad4510c1a2613e
Yuling Gu
admin_assigned
2024-11-25T21:42:05.890Z
62a8b91c1277a49db0c5c080
/avatars/9cf834a02b67af862659c9b9d4367a52.svg
Yuling Gu
user
SaveBertAndGpt
674414c8d3ad4510c1a2613f
Saumya Malik
admin_assigned
2024-11-25T21:42:11.222Z
63fee2851193ba2c22a1b687
/avatars/d2357740e01dafb216db0ed2b74a8100.svg
Saumya Malik
user
saumyamalik
674414c8d3ad4510c1a26140
Victoria Graf
admin_assigned
2024-11-25T21:42:17.089Z
646d699ed720ccd8cae5c011
/avatars/2e5ccee268081b7ba663eb6932552345.svg
Victoria Graf
user
VGraf
674414c8d3ad4510c1a26141
Jena D. Hwang
admin_assigned
2024-11-25T21:42:25.494Z
65c157eaf9957360824b0d87
https://cdn-avatars.huggingface.co/v1/production/uploads/65c157eaf9957360824b0d87/c1NWTKsFLATJLR7W6Kmqz.jpeg
jena hwang
user
jenah
674414c8d3ad4510c1a26142
Jiangjiang Yang
674414c8d3ad4510c1a26143
Ronan Le Bras
admin_assigned
2024-11-25T21:42:39.456Z
635049104e753c9940fefd71
https://cdn-avatars.huggingface.co/v1/production/uploads/635049104e753c9940fefd71/HgR43XIFw3dneY5ufrAE8.jpeg
Ronan Le Bras
user
ronanlb
674414c8d3ad4510c1a26144
Oyvind Tafjord
admin_assigned
2024-11-25T21:42:44.624Z
609b1aeff7c1303cfd1760f3
/avatars/3cb448514885656466886d7c84e12af1.svg
Oyvind Tafjord
user
OyvindTafjord
674414c8d3ad4510c1a26145
Chris Wilhelm
admin_assigned
2024-11-25T21:42:51.192Z
62bf74ca34afe69e1a1f98e3
/avatars/75aa4e89ea19a971300a57a208c686a6.svg
Chris Wilhelm
user
cmwilhelm
674414c8d3ad4510c1a26146
Luca Soldaini
admin_assigned
2024-11-25T21:42:58.621Z
5f04d8c45d08220171a0ad32
https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/uXEta6nqBabrUlAOXnS5g.jpeg
Luca Soldaini
user
soldni
674414c8d3ad4510c1a26147
Noah A. Smith
674414c8d3ad4510c1a26148
Yizhong Wang
admin_assigned
2024-11-25T21:43:10.147Z
6269be67d1ac0cde592aba29
https://cdn-avatars.huggingface.co/v1/production/uploads/6269be67d1ac0cde592aba29/EP3zZbD7-jWK9ITvHGbnZ.jpeg
Yizhong Wang
user
yizhongw
674414c8d3ad4510c1a26149
Pradeep Dasigi
admin_assigned
2024-11-25T21:43:15.425Z
6408fcc93461c51cf735a61e
/avatars/619f3653911d111f046a5a6c30fc8319.svg
Pradeep Dasigi
user
pradeepd
674414c8d3ad4510c1a2614a
Hannaneh Hajishirzi
TÜLU 3: Pushing Frontiers in Open Language Model Post-Training
Language model post-training is applied to refine behaviors and unlock new
skills across a wide range of recent language models, but open recipes for
applying these techniques lag behind proprietary ones. The underlying training
data and recipes for post-training are simultaneously the most important pieces
of the puzzle and the portion with the least transparency. To bridge this gap,
we introduce T\"ULU 3, a family of fully-open state-of-the-art post-trained
models, alongside its data, code, and training recipes, serving as a
comprehensive guide for modern post-training techniques. T\"ULU 3, which builds
on Llama 3.1 base models, achieves results surpassing the instruct versions of
Llama 3.1, Qwen 2.5, Mistral, and even closed models such as GPT-4o-mini and
Claude 3.5-Haiku. The training algorithms for our models include supervised
finetuning (SFT), Direct Preference Optimization (DPO), and a novel method we
call Reinforcement Learning with Verifiable Rewards (RLVR). With T\"ULU 3, we
introduce a multi-task evaluation scheme for post-training recipes with
development and unseen evaluations, standard benchmark implementations, and
substantial decontamination of existing open datasets on said benchmarks. We
conclude with analysis and discussion of training methods that did not reliably
improve performance.
  In addition to the T\"ULU 3 model weights and demo, we release the complete
recipe -- including datasets for diverse core skills, a robust toolkit for data
curation and evaluation, the training code and infrastructure, and, most
importantly, a detailed report for reproducing and further adapting the T\"ULU
3 approach to more domains.
T\"ULU 3, an open-source family of post-trained language models, introduces transparent training data, recipes, and advanced techniques to match or surpass proprietary models in performance.
supervised finetuning (SFT) Direct Preference Optimization (DPO) Reinforcement Learning with Verifiable Rewards (RLVR) multi-task evaluation scheme standard benchmark implementations data decontamination