Tuna: Instruction Tuning using Feedback from Large Language Models
Instruction tuning of open-source large language models (LLMs) like LLaMA,
using direct outputs from more powerful LLMs such as Instruct-GPT and GPT-4,
has proven to be a cost-effective way to align model behaviors with human
preferences. However, the instruction-tuned model has only seen one response
per instruction, lacking the knowledge of potentially better responses. In this
paper, we propose finetuning an instruction-tuned LLM using our novel
probabilistic ranking and contextual ranking approaches to
increase the likelihood of generating better responses. Probabilistic ranking
enables the instruction-tuned model to inherit the relative rankings of
high-quality and low-quality responses from the teacher LLM. On the other hand,
learning with contextual ranking allows the model to refine its own response
distribution using the contextual understanding ability of stronger LLMs.
Furthermore, we apply probabilistic ranking and contextual ranking sequentially
to the instruction-tuned LLM. The resulting model, which we call Tuna,
consistently improves the performance on Super Natural Instructions (119 test
tasks), LMentry (25 test tasks), Vicuna QA, and can even obtain better results
than several strong reinforcement learning baselines. Our code and data are
available at  https://github.com/microsoft/LMOps.
6535edac0917932040c4c475
Haoran Li
6535edac0917932040c4c476
Yiran Liu
admin_assigned
2023-10-23T08:47:42.148Z
64ba2a5962c0bfc5739b0661
/avatars/6c7b8495af6e7252de36b6e4071ca0a1.svg
Yiran Liu
user
liu-yr21
6535edac0917932040c4c477
Xingxing Zhang
admin_assigned
2023-10-23T08:47:51.466Z
64abbcff6cadc7aca584f71b
/avatars/fc6e85ad4a8befd133a37b411712c648.svg
Xingxing Zhang
user
THU-CHUNXIA
6535edac0917932040c4c478
Wei Lu
6535edac0917932040c4c479
Furu Wei
admin_assigned
2023-10-23T08:48:13.294Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
Tuna: Instruction Tuning using Feedback from Large Language Models
Instruction tuning of open-source large language models (LLMs) like LLaMA,
using direct outputs from more powerful LLMs such as Instruct-GPT and GPT-4,
has proven to be a cost-effective way to align model behaviors with human
preferences. However, the instruction-tuned model has only seen one response
per instruction, lacking the knowledge of potentially better responses. In this
paper, we propose finetuning an instruction-tuned LLM using our novel
probabilistic ranking and contextual ranking approaches to
increase the likelihood of generating better responses. Probabilistic ranking
enables the instruction-tuned model to inherit the relative rankings of
high-quality and low-quality responses from the teacher LLM. On the other hand,
learning with contextual ranking allows the model to refine its own response
distribution using the contextual understanding ability of stronger LLMs.
Furthermore, we apply probabilistic ranking and contextual ranking sequentially
to the instruction-tuned LLM. The resulting model, which we call Tuna,
consistently improves the performance on Super Natural Instructions (119 test
tasks), LMentry (25 test tasks), Vicuna QA, and can even obtain better results
than several strong reinforcement learning baselines. Our code and data are
available at  https://github.com/microsoft/LMOps.
A novel finetuning method using probabilistic and contextual ranking improves instruction-tuned LLM performance across multiple benchmarks by leveraging stronger teacher models.
probabilistic ranking contextual ranking instruction-tuned LLM fine-tuning Super Natural Instructions LMentry Vicuna QA reinforcement learning