GenARM: Reward Guided Generation with Autoregressive Reward Model for
  Test-time Alignment
Large Language Models (LLMs) exhibit impressive capabilities but require
careful alignment with human preferences. Traditional training-time methods
finetune LLMs using human preference datasets but incur significant training
costs and require repeated training to handle diverse user preferences.
Test-time alignment methods address this by using reward models (RMs) to guide
frozen LLMs without retraining. However, existing test-time approaches rely on
trajectory-level RMs which are designed to evaluate complete responses, making
them unsuitable for autoregressive text generation that requires computing
next-token rewards from partial responses. To address this, we introduce
GenARM, a test-time alignment approach that leverages the Autoregressive Reward
Model--a novel reward parametrization designed to predict next-token rewards
for efficient and effective autoregressive generation. Theoretically, we
demonstrate that this parametrization can provably guide frozen LLMs toward any
distribution achievable by traditional RMs within the KL-regularized
reinforcement learning framework. Experimental results show that GenARM
significantly outperforms prior test-time alignment baselines and matches the
performance of training-time methods. Additionally, GenARM enables efficient
weak-to-strong guidance, aligning larger LLMs with smaller RMs without the high
costs of training larger models. Furthermore, GenARM supports multi-objective
alignment, allowing real-time trade-offs between preference dimensions and
catering to diverse user preferences without retraining.
6708bba1d2dd526a476139ad
Yuancheng Xu
claimed_verified
2024-10-15T09:11:25.869Z
6505e2cdad3134ed7e63c8d0
/avatars/f0e5dcf5a3deb96d2c9cb50b4433cc13.svg
Yuancheng Xu
user
YuanchengXu
6708bba1d2dd526a476139ae
Udari Madhushani Sehwag
6708bba1d2dd526a476139af
Alec Koppel
6708bba1d2dd526a476139b0
Sicheng Zhu
6708bba1d2dd526a476139b1
Bang An
extracted_confirmed
2024-10-11T14:15:10.381Z
645071555fdcff143ad8e07f
/avatars/a59f4e518a500e1c58b3a6d688644432.svg
Bang An
user
bangan
6708bba1d2dd526a476139b2
Furong Huang
extracted_pending
2024-10-11T05:46:10.318Z
64cbc3e2a257a3212c00a115
/avatars/836e61be4aeda2080ddf2db9f2626cc6.svg
Furong Huang Lab at UMD
user
furongh-lab
6708bba1d2dd526a476139b3
Sumitra Ganesh
GenARM: Reward Guided Generation with Autoregressive Reward Model for
  Test-time Alignment
Large Language Models (LLMs) exhibit impressive capabilities but require
careful alignment with human preferences. Traditional training-time methods
finetune LLMs using human preference datasets but incur significant training
costs and require repeated training to handle diverse user preferences.
Test-time alignment methods address this by using reward models (RMs) to guide
frozen LLMs without retraining. However, existing test-time approaches rely on
trajectory-level RMs which are designed to evaluate complete responses, making
them unsuitable for autoregressive text generation that requires computing
next-token rewards from partial responses. To address this, we introduce
GenARM, a test-time alignment approach that leverages the Autoregressive Reward
Model--a novel reward parametrization designed to predict next-token rewards
for efficient and effective autoregressive generation. Theoretically, we
demonstrate that this parametrization can provably guide frozen LLMs toward any
distribution achievable by traditional RMs within the KL-regularized
reinforcement learning framework. Experimental results show that GenARM
significantly outperforms prior test-time alignment baselines and matches the
performance of training-time methods. Additionally, GenARM enables efficient
weak-to-strong guidance, aligning larger LLMs with smaller RMs without the high
costs of training larger models. Furthermore, GenARM supports multi-objective
alignment, allowing real-time trade-offs between preference dimensions and
catering to diverse user preferences without retraining.
Large Language Models (LLMs) human preference datasets reward models (RMs) Autoregressive Reward Model next-token rewards autoregressive generation KL-regularized reinforcement learning multi-objective alignment