Beyond Human Data: Scaling Self-Training for Problem-Solving with
  Language Models
Fine-tuning language models~(LMs) on human-generated data remains a prevalent
practice. However, the performance of such models is often limited by the
quantity and diversity of high-quality human data. In this paper, we explore
whether we can go beyond human data on tasks where we have access to scalar
feedback, for example, on math problems where one can verify correctness. To do
so, we investigate a simple self-training method based on
expectation-maximization, which we call ReST^{EM}, where we (1) generate
samples from the model and filter them using binary feedback, (2) fine-tune the
model on these samples, and (3) repeat this process a few times. Testing on
advanced MATH reasoning and APPS coding benchmarks using PaLM-2 models, we find
that ReST^{EM} scales favorably with model size and significantly surpasses
fine-tuning only on human data. Overall, our findings suggest self-training
with feedback can substantially reduce dependence on human-generated data.
6577f1e4ee33d547aebf898d
Avi Singh
6577f1e4ee33d547aebf898e
John D. Co-Reyes
6577f1e4ee33d547aebf898f
Rishabh Agarwal
extracted_pending
2023-12-12T05:38:44.332Z
647875e37b370854241b89a8
/avatars/fd25f7e5960e4e132b9ee689f114820b.svg
Rishabh Agarwal
user
agarwl
6577f1e4ee33d547aebf8990
Ankesh Anand
admin_assigned
2023-12-12T08:23:09.319Z
626321a613cc9ec91bd7de48
/avatars/79e4f1843a5a8041197b01772f21bc48.svg
Ankesh Anand
user
ankeshanand
6577f1e4ee33d547aebf8991
Piyush Patil
6577f1e4ee33d547aebf8992
Peter J. Liu
admin_assigned
2023-12-12T08:23:45.836Z
62ded294dcdc9043efaa4fc3
/avatars/1294078b8cd5c1901e4021253d877357.svg
Peter J Liu
user
peterjliu
6577f1e4ee33d547aebf8993
James Harrison
6577f1e4ee33d547aebf8994
Jaehoon Lee
6577f1e4ee33d547aebf8995
Kelvin Xu
6577f1e4ee33d547aebf8996
Aaron Parisi
admin_assigned
2023-12-12T08:25:14.232Z
635f493ba81c7f7424a702ae
/avatars/6f05d4f76609607966ed4635cb1e0a02.svg
Aaron Parisi
user
AnotherA
6577f1e4ee33d547aebf8997
Abhishek Kumar
6577f1e4ee33d547aebf8998
Alex Alemi
admin_assigned
2023-12-12T08:25:41.874Z
6303ff84fc783bfc74441afc
/avatars/bc295e0d94c9c8be0c6a8264587ad4a5.svg
Alex Alemi
user
knowflake
6577f1e4ee33d547aebf8999
Alex Rizkowsky
6577f1e4ee33d547aebf899a
Azade Nova
6577f1e4ee33d547aebf899b
Ben Adlam
6577f1e4ee33d547aebf899c
Bernd Bohnet
6577f1e4ee33d547aebf899d
Hanie Sedghi
6577f1e4ee33d547aebf899e
Igor Mordatch
6577f1e4ee33d547aebf899f
Isabelle Simpson
6577f1e4ee33d547aebf89a0
Izzeddin Gur
admin_assigned
2023-12-12T08:26:12.768Z
64c0ccc19832a455bc2a2069
/avatars/9d4fdf13cd62c6c258b5f50d67c8a1b1.svg
IZZEDDIN GUR
user
izzeddingur
6577f1e4ee33d547aebf89a1
Jasper Snoek
6577f1e4ee33d547aebf89a2
Jeffrey Pennington
6577f1e4ee33d547aebf89a3
Jiri Hron
admin_assigned
2023-12-12T08:26:27.286Z
62af5f9e24488e6adfa86321
/avatars/5990e5f656c75530e6e8f01f644f7c8d.svg
Jiri Hron
user
jirihron
6577f1e4ee33d547aebf89a4
Kathleen Kenealy
6577f1e4ee33d547aebf89a5
Kevin Swersky
admin_assigned
2023-12-12T08:26:41.648Z
630e6ef664f1f8d0c771b758
/avatars/1df2bfadb2b6fdf8307189936efc6ef0.svg
Kevin Swersky
user
kswersky
6577f1e4ee33d547aebf89a6
Kshiteej Mahajan
6577f1e4ee33d547aebf89a7
Laura Culp
6577f1e4ee33d547aebf89a8
Lechao Xiao
6577f1e4ee33d547aebf89a9
Maxwell L. Bileschi
6577f1e4ee33d547aebf89aa
Noah Constant
6577f1e4ee33d547aebf89ab
Roman Novak
admin_assigned
2023-12-12T08:27:20.964Z
63e2769bf0740bec2bfff1a1
/avatars/e2841a836319d43035bcddfbfe3344d0.svg
Roman Novak
user
kulpims
6577f1e4ee33d547aebf89ac
Rosanne Liu
6577f1e4ee33d547aebf89ad
Tris Warkentin
admin_assigned
2023-12-12T08:27:44.017Z
64d51be85e5f05485ca50751
/avatars/5802fcc5a5f1e089df41232892f8f502.svg
Tris Warkentin
user
trisfromgoogle
6577f1e4ee33d547aebf89ae
Yundi Qian
6577f1e4ee33d547aebf89af
Ethan Dyer
6577f1e4ee33d547aebf89b0
Behnam Neyshabur
6577f1e4ee33d547aebf89b1
Jascha Sohl-Dickstein
admin_assigned
2023-12-12T08:28:09.903Z
63d209ae8eaa4831006b3b0d
/avatars/51cc0f6dfab2233bddd73bfcd0c8aee3.svg
Jascha Sohl-Dickstein
user
jaschasd
6577f1e4ee33d547aebf89b2
Noah Fiedel
admin_assigned
2023-12-12T08:28:18.023Z
62f82ebd870a3f98bbf9e62a
/avatars/3b03299e128abf2c8ba82db704855de7.svg
Noah Fiedel
user
nfiedel
Beyond Human Data: Scaling Self-Training for Problem-Solving with
  Language Models
Fine-tuning language models~(LMs) on human-generated data remains a prevalent
practice. However, the performance of such models is often limited by the
quantity and diversity of high-quality human data. In this paper, we explore
whether we can go beyond human data on tasks where we have access to scalar
feedback, for example, on math problems where one can verify correctness. To do
so, we investigate a simple self-training method based on
expectation-maximization, which we call ReST^{EM}, where we (1) generate
samples from the model and filter them using binary feedback, (2) fine-tune the
model on these samples, and (3) repeat this process a few times. Testing on
advanced MATH reasoning and APPS coding benchmarks using PaLM-2 models, we find
that ReST^{EM} scales favorably with model size and significantly surpasses
fine-tuning only on human data. Overall, our findings suggest self-training
with feedback can substantially reduce dependence on human-generated data.
Self-training with scalar feedback using expectation-maximization improves language model performance on tasks with verifiable answers, reducing reliance on human data.
self-training expectation-maximization scalar feedback MATH reasoning APPS coding benchmarks fine-tuning