REFINE-AF: A Task-Agnostic Framework to Align Language Models via
  Self-Generated Instructions using Reinforcement Learning from Automated
  Feedback
Instruction-based Large Language Models (LLMs) have proven effective in
numerous few-shot or zero-shot Natural Language Processing (NLP) tasks.
However, creating human-annotated instruction data is time-consuming,
expensive, and often limited in quantity and task diversity. Previous research
endeavors have attempted to address this challenge by proposing frameworks
capable of generating instructions in a semi-automated and task-agnostic manner
directly from the model itself. Many of these efforts have relied on large
API-only parameter-based models such as GPT-3.5 (175B), which are expensive,
and subject to limits on a number of queries. This paper explores the
performance of three open-source small LLMs such as LLaMA 2-7B, LLama 2-13B,
and Mistral 7B, using a semi-automated framework, thereby reducing human
intervention, effort, and cost required to generate an instruction dataset for
fine-tuning LLMs. Furthermore, we demonstrate that incorporating a
Reinforcement Learning (RL) based training algorithm into this LLMs-based
framework leads to further enhancements. Our evaluation of the dataset reveals
that these RL-based frameworks achieve a substantial improvements in 63-66% of
the tasks compared to previous approaches.
6822a0f7f90521b5d8cb4573
Aniruddha Roy
6822a0f7f90521b5d8cb4574
Pretam Ray
admin_assigned
2025-05-13T08:32:17.365Z
62d3ac975a869f4338852b6f
/avatars/4094eb1ea1244b0a9b60344528efa25a.svg
Pretam Ray
user
Pretam
6822a0f7f90521b5d8cb4575
Abhilash Nandy
admin_assigned
2025-05-13T08:32:23.396Z
5f89da6c5d083370c711f37c
/avatars/c2f17a4a636973817fd5da2ae6dbaac3.svg
Abhilash Nandy
user
abhi1nandy2
6822a0f7f90521b5d8cb4576
Somak Aditya
6822a0f7f90521b5d8cb4577
Pawan Goyal
REFINE-AF: A Task-Agnostic Framework to Align Language Models via
  Self-Generated Instructions using Reinforcement Learning from Automated
  Feedback
Instruction-based Large Language Models (LLMs) have proven effective in
numerous few-shot or zero-shot Natural Language Processing (NLP) tasks.
However, creating human-annotated instruction data is time-consuming,
expensive, and often limited in quantity and task diversity. Previous research
endeavors have attempted to address this challenge by proposing frameworks
capable of generating instructions in a semi-automated and task-agnostic manner
directly from the model itself. Many of these efforts have relied on large
API-only parameter-based models such as GPT-3.5 (175B), which are expensive,
and subject to limits on a number of queries. This paper explores the
performance of three open-source small LLMs such as LLaMA 2-7B, LLama 2-13B,
and Mistral 7B, using a semi-automated framework, thereby reducing human
intervention, effort, and cost required to generate an instruction dataset for
fine-tuning LLMs. Furthermore, we demonstrate that incorporating a
Reinforcement Learning (RL) based training algorithm into this LLMs-based
framework leads to further enhancements. Our evaluation of the dataset reveals
that these RL-based frameworks achieve a substantial improvements in 63-66% of
the tasks compared to previous approaches.
Semi-automated frameworks using open-source small LLMs and reinforcement learning significantly improve instruction dataset generation for LLM fine-tuning across various tasks.
Instruction-based Large Language Models (LLMs) Natural Language Processing (NLP) few-shot zero-shot semi-automated framework task-agnostic GPT-3.5 LLaMA 2-7B LLama 2-13B Mistral 7B Reinforcement Learning (RL) fine-tuning