Self-play with Execution Feedback: Improving Instruction-following
  Capabilities of Large Language Models
One core capability of large language models (LLMs) is to follow natural
language instructions. However, the issue of automatically constructing
high-quality training data to enhance the complex instruction-following
abilities of LLMs without manual annotation remains unresolved. In this paper,
we introduce AutoIF, the first scalable and reliable method for automatically
generating instruction-following training data. AutoIF transforms the
validation of instruction-following data quality into code verification,
requiring LLMs to generate instructions, the corresponding code to check the
correctness of the instruction responses, and unit test samples to verify the
code's correctness. Then, execution feedback-based rejection sampling can
generate data for Supervised Fine-Tuning (SFT) and Reinforcement Learning from
Human Feedback (RLHF) training. AutoIF achieves significant improvements across
three training algorithms, SFT, Offline DPO, and Online DPO, when applied to
the top open-source LLMs, Qwen2 and LLaMA3, in self-alignment and
strong-to-weak distillation settings. Our code is publicly available at
https://github.com/QwenLM/AutoIF.
66755c0fd5ac75b591f746da
Guanting Dong
admin_assigned
2024-06-21T12:22:03.427Z
61cd4b833dd34ba1985e0753
https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
KABI
user
dongguanting
66755c0fd5ac75b591f746db
Keming Lu
admin_assigned
2024-06-21T12:22:09.957Z
6453fa96ed6d7fede94408e0
/avatars/e8c9025ef24cec958c87a1008bb54fd7.svg
Keming Lu
user
keminglu
66755c0fd5ac75b591f746dc
Chengpeng Li
admin_assigned
2024-06-21T12:22:19.159Z
65294b334d7cf551ac50d6a6
/avatars/75d21e20b711b871616ef3850bb900b7.svg
ChengpengLi
user
ChengpengLi
66755c0fd5ac75b591f746dd
Tingyu Xia
claimed_verified
2024-10-16T09:44:54.377Z
653b5cb0d994e992e2831539
/avatars/e6b917cb263f10a6d9c72ec539276428.svg
xiatingyu
user
xiatingyu
66755c0fd5ac75b591f746de
Bowen Yu
claimed_verified
2024-06-24T07:32:26.389Z
63d9d68c1cae35c27bf7a6a7
/avatars/b5ad98cf269ae5f1fe90861fb4170fae.svg
Bowen Yu
user
Tigerph
66755c0fd5ac75b591f746df
Chang Zhou
66755c0fd5ac75b591f746e0
Jingren Zhou
admin_assigned
2024-06-21T12:22:56.911Z
602f88f5e8149a962412a667
/avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
Zhou
user
Jingren
Self-play with Execution Feedback: Improving Instruction-following
  Capabilities of Large Language Models
One core capability of large language models (LLMs) is to follow natural
language instructions. However, the issue of automatically constructing
high-quality training data to enhance the complex instruction-following
abilities of LLMs without manual annotation remains unresolved. In this paper,
we introduce AutoIF, the first scalable and reliable method for automatically
generating instruction-following training data. AutoIF transforms the
validation of instruction-following data quality into code verification,
requiring LLMs to generate instructions, the corresponding code to check the
correctness of the instruction responses, and unit test samples to verify the
code's correctness. Then, execution feedback-based rejection sampling can
generate data for Supervised Fine-Tuning (SFT) and Reinforcement Learning from
Human Feedback (RLHF) training. AutoIF achieves significant improvements across
three training algorithms, SFT, Offline DPO, and Online DPO, when applied to
the top open-source LLMs, Qwen2 and LLaMA3, in self-alignment and
strong-to-weak distillation settings. Our code is publicly available at
https://github.com/QwenLM/AutoIF.
AutoIF is a scalable method for automatically generating high-quality instruction-following training data by using code verification and execution feedback-based rejection sampling, improving various training algorithms for LLMs.
Large Language Models (LLMs) instruction-following training data code verification execution feedback-based rejection sampling Supervised Fine-Tuning (SFT) Reinforcement Learning from Human Feedback (RLHF) Offline DPO Online DPO Qwen2 LLaMA3 self-alignment strong-to-weak distillation