Response Tuning: Aligning Large Language Models without Instruction
Instruction tuning-supervised fine-tuning using instruction-response pairs-is
a foundational step in transitioning pre-trained Large Language Models (LLMs)
into helpful and safe chat assistants. Our hypothesis is that establishing an
adequate output space can enable such a transition given the capabilities
inherent in pre-trained LLMs. To verify this, we propose Response Tuning (RT),
which eliminates the instruction-conditioning step in instruction tuning and
solely focuses on response space supervision. Our experiments demonstrate that
RT models, trained only using responses, can effectively respond to a wide
range of instructions and exhibit helpfulness comparable to that of their
instruction-tuned counterparts. Furthermore, we observe that controlling the
training response distribution can significantly improve their user preference
or elicit target behaviors such as refusing assistance for unsafe queries. Our
findings illuminate the role of establishing an adequate output space in
alignment, highlighting the potential of the extensive inherent capabilities of
pre-trained LLMs.
6707b25a40b1f23d30085d85
Seokhyun An
claimed_verified
2024-10-10T10:59:05.265Z
617017b7544a1c62b298b87d
/avatars/a569d976eb26807bd126de63a6b808fd.svg
Seokhyun An
user
seokhyun
6707b25a40b1f23d30085d86
Hyounghun Kim
claimed_verified
2024-10-10T14:03:08.497Z
64d7162b3ca2924d6e8593e4
/avatars/51b28e4652bbb9d35adbd62aece59cd0.svg
Hyounghun Kim
user
khh3323
Response Tuning: Aligning Large Language Models without Instruction
Instruction tuning-supervised fine-tuning using instruction-response pairs-is
a foundational step in transitioning pre-trained Large Language Models (LLMs)
into helpful and safe chat assistants. Our hypothesis is that establishing an
adequate output space can enable such a transition given the capabilities
inherent in pre-trained LLMs. To verify this, we propose Response Tuning (RT),
which eliminates the instruction-conditioning step in instruction tuning and
solely focuses on response space supervision. Our experiments demonstrate that
RT models, trained only using responses, can effectively respond to a wide
range of instructions and exhibit helpfulness comparable to that of their
instruction-tuned counterparts. Furthermore, we observe that controlling the
training response distribution can significantly improve their user preference
or elicit target behaviors such as refusing assistance for unsafe queries. Our
findings illuminate the role of establishing an adequate output space in
alignment, highlighting the potential of the extensive inherent capabilities of
pre-trained LLMs.
Response Tuning, an approach focusing on response space supervision without instruction conditioning, effectively transitions pre-trained LLMs into helpful and safe chat assistants.
instruction tuning instruction-response pairs Large Language Models (LLMs) Response Tuning (RT) response space supervision user preference target behaviors alignment capabilities