A Common Pitfall of Margin-based Language Model Alignment: Gradient
  Entanglement
Reinforcement Learning from Human Feedback (RLHF) has become the predominant
approach for language model (LM) alignment. At its core, RLHF uses a
margin-based loss for preference optimization, specifying ideal LM behavior
only by the difference between preferred and dispreferred responses. In this
paper, we identify a common pitfall of margin-based methods -- the
under-specification of ideal LM behavior on preferred and dispreferred
responses individually, which leads to two unintended consequences as the
margin increases: (1) The probability of dispreferred (e.g., unsafe) responses
may increase, resulting in potential safety alignment failures. (2) The
probability of preferred responses may decrease, even when those responses are
ideal. We demystify the reasons behind these problematic behaviors:
margin-based losses couple the change in the preferred probability to the
gradient of the dispreferred one, and vice versa, often preventing the
preferred probability from increasing while the dispreferred one decreases, and
thus causing a synchronized increase or decrease in both probabilities. We term
this effect, inherent in margin-based objectives, gradient entanglement.
Formally, we derive conditions for general margin-based alignment objectives
under which gradient entanglement becomes concerning: the inner product of the
gradients of preferred and dispreferred log-probabilities is large relative to
the individual gradient norms. We theoretically investigate why such inner
products can be large when aligning language models and empirically validate
our findings. Empirical implications of our framework extend to explaining
important differences in the training dynamics of various preference
optimization algorithms, and suggesting potential algorithm designs to mitigate
the under-specification issue of margin-based methods and thereby improving
language model alignment.
6711dffcff45f0171ce2a393
Hui Yuan
extracted_pending
2024-10-18T04:11:40.965Z
65256f6ec6a6a53e2e693fc7
/avatars/928f29207c196bbffd53bb3b4a3bfd17.svg
Hui Yuan
user
huiyuan23
6711dffcff45f0171ce2a394
Yifan Zeng
claimed_verified
2024-10-18T08:09:54.399Z
6245285af59b8d262df3321b
https://cdn-avatars.huggingface.co/v1/production/uploads/6245285af59b8d262df3321b/dvy__dTf-miJ60IbveDg4.jpeg
Yifan Zeng
user
yokey
6711dffcff45f0171ce2a395
Yue Wu
6711dffcff45f0171ce2a396
Huazheng Wang
claimed_verified
2025-07-30T09:05:46.292Z
688969c4d0ac71994caa0df6
/avatars/f8f89e0ca810908cfe21c921d8373e28.svg
Huazheng Wang
user
huazhengwang
6711dffcff45f0171ce2a397
Mengdi Wang
admin_assigned
2024-10-21T12:17:19.140Z
6599415e8c8ac79295e0b5e3
/avatars/85500bc8d2cd51444adcc19b1f8db313.svg
Mengdi Wang
user
Edify-Kd2024
6711dffcff45f0171ce2a398
Liu Leqi
admin_assigned
2024-10-21T12:18:14.977Z
62791751bba7c62cbab1076e
/avatars/9efb4f809d8961b47d3ae687f6e8dc2f.svg
Liu Leqi
user
leqiliu
A Common Pitfall of Margin-based Language Model Alignment: Gradient
  Entanglement
Reinforcement Learning from Human Feedback (RLHF) has become the predominant
approach for language model (LM) alignment. At its core, RLHF uses a
margin-based loss for preference optimization, specifying ideal LM behavior
only by the difference between preferred and dispreferred responses. In this
paper, we identify a common pitfall of margin-based methods -- the
under-specification of ideal LM behavior on preferred and dispreferred
responses individually, which leads to two unintended consequences as the
margin increases: (1) The probability of dispreferred (e.g., unsafe) responses
may increase, resulting in potential safety alignment failures. (2) The
probability of preferred responses may decrease, even when those responses are
ideal. We demystify the reasons behind these problematic behaviors:
margin-based losses couple the change in the preferred probability to the
gradient of the dispreferred one, and vice versa, often preventing the
preferred probability from increasing while the dispreferred one decreases, and
thus causing a synchronized increase or decrease in both probabilities. We term
this effect, inherent in margin-based objectives, gradient entanglement.
Formally, we derive conditions for general margin-based alignment objectives
under which gradient entanglement becomes concerning: the inner product of the
gradients of preferred and dispreferred log-probabilities is large relative to
the individual gradient norms. We theoretically investigate why such inner
products can be large when aligning language models and empirically validate
our findings. Empirical implications of our framework extend to explaining
important differences in the training dynamics of various preference
optimization algorithms, and suggesting potential algorithm designs to mitigate
the under-specification issue of margin-based methods and thereby improving
language model alignment.
RLHF with margin-based methods can lead to increased dispreferred responses and decreased preferred responses due to gradient entanglement, complicating model alignment.
Reinforcement Learning from Human Feedback (RLHF) margin-based loss preference optimization language model alignment gradient entanglement log-probabilities gradient norms