PERL: Parameter Efficient Reinforcement Learning from Human Feedback
Reinforcement Learning from Human Feedback (RLHF) has proven to be a strong
method to align Pretrained Large Language Models (LLMs) with human preferences.
But training models with RLHF is computationally expensive, and an overall
complex process. In this work, we study RLHF where the underlying models are
trained using the parameter efficient method of Low-Rank Adaptation (LoRA)
introduced by Hu et al. [2021]. We investigate the setup of "Parameter
Efficient Reinforcement Learning" (PERL), in which we perform reward model
training and reinforcement learning using LoRA. We compare PERL to conventional
fine-tuning (full-tuning) across various configurations for 7 benchmarks,
including 2 novel datasets, of reward modeling and reinforcement learning. We
find that PERL performs on par with the conventional RLHF setting, while
training faster, and with less memory. This enables the high performance of
RLHF, while reducing the computational burden that limits its adoption as an
alignment technique for Large Language Models. We also release 2 novel thumbs
up/down preference datasets: "Taskmaster Coffee", and "Taskmaster Ticketing" to
promote research around RLHF.
65f91d16ea2d72e3b584f191
Hakim Sidahmed
65f91d16ea2d72e3b584f192
Samrat Phatale
admin_assigned
2024-03-19T09:30:47.424Z
64fb65f348411fc78979a654
/avatars/66b61203f2a6fd07f9026ba97f9e8c5d.svg
Samrat Phatale
user
samratph
65f91d16ea2d72e3b584f193
Alex Hutcheson
admin_assigned
2024-03-19T09:30:53.475Z
65e8e8213ef9a491d42d7f36
/avatars/35e63bfd293ee4a31f4057122c403ed0.svg
alexhutcheson
user
alexhutcheson
65f91d16ea2d72e3b584f194
Zhuonan Lin
claimed_verified
2024-03-19T20:44:30.657Z
65f9ef2b3260327fc7033467
/avatars/623ef7c11543b5009cb2d6b8ad2e082f.svg
Zhuonan Lin
user
zhuonanlin
65f91d16ea2d72e3b584f195
Zhang Chen
65f91d16ea2d72e3b584f196
Zac Yu
admin_assigned
2024-03-19T09:31:43.363Z
6436815a0b2c0d86d0b183fc
/avatars/fc9d7825e5f108f0ec231c9f4d1caeb6.svg
zac yu
user
zacyy
65f91d16ea2d72e3b584f197
Jarvis Jin
65f91d16ea2d72e3b584f198
Roman Komarytsia
65f91d16ea2d72e3b584f199
Christiane Ahlheim
65f91d16ea2d72e3b584f19a
Yonghao Zhu
65f91d16ea2d72e3b584f19b
Simral Chaudhary
claimed_verified
2024-03-21T08:27:53.457Z
65fa130087d1c912d9944f4e
/avatars/8976decb5c7706e8007d707d655cd82a.svg
Simral Chaudhary
user
simral
65f91d16ea2d72e3b584f19c
Bowen Li
claimed_verified
2024-03-25T09:44:20.411Z
65a8b8689db09e1fc1502b7f
/avatars/bba2faab9c040a7aba40c53773b37e21.svg
Bowen Li
user
bowenlee919
65f91d16ea2d72e3b584f19d
Saravanan Ganesh
admin_assigned
2024-03-19T09:32:51.302Z
6450077817b6189cda58fbcd
/avatars/df47f880b3d7c824b1a616c39ebdfb1c.svg
Saravanan Ganesh
user
srrvnn
65f91d16ea2d72e3b584f19e
Bill Byrne
65f91d16ea2d72e3b584f19f
Jessica Hoffmann
admin_assigned
2024-03-19T09:33:03.861Z
6429734a8136224fee06a749
/avatars/dafe1e85f3d97ab9acd5547fd2919ba9.svg
Jessica Hoffmann
user
jessicahoffmann111
65f91d16ea2d72e3b584f1a0
Hassan Mansoor
admin_assigned
2024-03-19T09:33:09.298Z
629098fd5463575364e7697a
/avatars/e45f3ee28bff3571f76caf847d3c36db.svg
Hassan Mansoor
user
HassanMansoor
65f91d16ea2d72e3b584f1a1
Wei Li
65f91d16ea2d72e3b584f1a2
Abhinav Rastogi
claimed_verified
2024-03-26T07:55:13.485Z
653966932a101a151da1155c
/avatars/10baa6536793b397c05028011d4ffcdc.svg
Abhinav Rastogi
user
abhirast
65f91d16ea2d72e3b584f1a3
Lucas Dixon
admin_assigned
2024-03-19T09:33:50.021Z
64c25a9c2b1ffc2eecbce31d
https://cdn-avatars.huggingface.co/v1/production/uploads/64c25a9c2b1ffc2eecbce31d/uTp3yOPdmCSAmhzWwKUjK.png
Lucas Dixon
user
iislucas
PERL: Parameter Efficient Reinforcement Learning from Human Feedback
Reinforcement Learning from Human Feedback (RLHF) has proven to be a strong
method to align Pretrained Large Language Models (LLMs) with human preferences.
But training models with RLHF is computationally expensive, and an overall
complex process. In this work, we study RLHF where the underlying models are
trained using the parameter efficient method of Low-Rank Adaptation (LoRA)
introduced by Hu et al. [2021]. We investigate the setup of "Parameter
Efficient Reinforcement Learning" (PERL), in which we perform reward model
training and reinforcement learning using LoRA. We compare PERL to conventional
fine-tuning (full-tuning) across various configurations for 7 benchmarks,
including 2 novel datasets, of reward modeling and reinforcement learning. We
find that PERL performs on par with the conventional RLHF setting, while
training faster, and with less memory. This enables the high performance of
RLHF, while reducing the computational burden that limits its adoption as an
alignment technique for Large Language Models. We also release 2 novel thumbs
up/down preference datasets: "Taskmaster Coffee", and "Taskmaster Ticketing" to
promote research around RLHF.
Parameter-efficient reinforcement learning using LoRA in RLHF matches conventional RLHF performance with reduced computational cost and memory usage.
Reinforcement Learning from Human Feedback Pretrained Large Language Models Low-Rank Adaptation Parameter Efficient Reinforcement Learning reward model training reinforcement learning full-tuning Taskmaster Coffee Taskmaster Ticketing