Reward Steering with Evolutionary Heuristics for Decoding-time Alignment
The widespread applicability and increasing omnipresence of LLMs have
instigated a need to align LLM responses to user and stakeholder preferences.
Many preference optimization approaches have been proposed that fine-tune LLM
parameters to achieve good alignment. However, such parameter tuning is known
to interfere with model performance on many tasks. Moreover, keeping up with
shifting user preferences is tricky in such a situation. Decoding-time
alignment with reward model guidance solves these issues at the cost of
increased inference time. However, most of such methods fail to strike the
right balance between exploration and exploitation of reward -- often due to
the conflated formulation of these two aspects - to give well-aligned
responses. To remedy this we decouple these two aspects and implement them in
an evolutionary fashion: exploration is enforced by decoding from mutated
instructions and exploitation is represented as the periodic replacement of
poorly-rewarded generations with well-rewarded ones. Empirical evidences
indicate that this strategy outperforms many preference optimization and
decode-time alignment approaches on two widely accepted alignment benchmarks
AlpacaEval 2 and MT-Bench. Our implementation will be available at:
https://darwin-alignment.github.io.
6678f8029f2810b009e0463a
Chia-Yu Hung
claimed_verified
2024-06-24T07:27:34.077Z
62864cdf011a13fe5f84d746
https://cdn-avatars.huggingface.co/v1/production/uploads/62864cdf011a13fe5f84d746/5yPYgAtGJTaUq4fHhh9Pr.jpeg
Hung Chia Yu
user
hungchiayu
6678f8029f2810b009e0463b
Navonil Majumder
admin_assigned
2024-06-24T13:43:09.664Z
644e20f267c9458c912bd246
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5j7LZ-6sCarsEvKr2tPAj.png
Navonil Majumder
user
hollownight
6678f8029f2810b009e0463c
Ambuj Mehrish
admin_assigned
2024-06-24T13:43:15.789Z
6350a5170f376d3c482c9738
/avatars/1dcf14c35a5a2018879c9c94b3ebe345.svg
Ambuj Mehrish
user
ambujm22
6678f8029f2810b009e0463d
Soujanya Poria
claimed_verified
2024-06-24T13:10:35.414Z
626b626405fe1cb65725aca1
https://cdn-avatars.huggingface.co/v1/production/uploads/626b626405fe1cb65725aca1/ZVSbhynzpQhVGq9kGywW6.png
Soujanya Poria
user
soujanyaporia
Reward Steering with Evolutionary Heuristics for Decoding-time Alignment
The widespread applicability and increasing omnipresence of LLMs have
instigated a need to align LLM responses to user and stakeholder preferences.
Many preference optimization approaches have been proposed that fine-tune LLM
parameters to achieve good alignment. However, such parameter tuning is known
to interfere with model performance on many tasks. Moreover, keeping up with
shifting user preferences is tricky in such a situation. Decoding-time
alignment with reward model guidance solves these issues at the cost of
increased inference time. However, most of such methods fail to strike the
right balance between exploration and exploitation of reward -- often due to
the conflated formulation of these two aspects - to give well-aligned
responses. To remedy this we decouple these two aspects and implement them in
an evolutionary fashion: exploration is enforced by decoding from mutated
instructions and exploitation is represented as the periodic replacement of
poorly-rewarded generations with well-rewarded ones. Empirical evidences
indicate that this strategy outperforms many preference optimization and
decode-time alignment approaches on two widely accepted alignment benchmarks
AlpacaEval 2 and MT-Bench. Our implementation will be available at:
https://darwin-alignment.github.io.
An evolutionary approach decouples exploration and exploitation in decoding-time alignment to enhance LLM preferences without compromising performance.
LLMs parameter tuning reward model decoding-time alignment evolutionary fashion mutated instructions poor-rewarded generations well-rewarded generations AlpacaEval 2 MT-Bench