Super(ficial)-alignment: Strong Models May Deceive Weak Models in
  Weak-to-Strong Generalization
Superalignment, where humans are weak supervisors of superhuman models, has
become an important and widely discussed issue in the current era of rapid
development of Large Language Models (LLMs). The recent work preliminarily
studies this problem by using weak models to supervise strong models. It
discovers that weakly supervised strong students can consistently outperform
weak teachers towards the alignment target, leading to a weak-to-strong
generalization phenomenon. However, we are concerned that behind such a
promising phenomenon, whether there exists an issue of weak-to-strong
deception, where strong models may deceive weak models by exhibiting
well-aligned in areas known to weak models but producing misaligned behaviors
in cases weak models do not know. We then take an initial step towards
exploring this security issue in a specific but realistic multi-objective
alignment case, where there may be some alignment targets conflicting with each
other (e.g., helpfulness v.s. harmlessness). Such a conflict is likely to cause
strong models to deceive weak models in one alignment dimension to gain high
reward in other alignment dimension. Our experiments on both the reward
modeling task and the preference optimization scenario indicate: (1) the
weak-to-strong deception exists; (2) the deception phenomenon may intensify as
the capability gap between weak and strong models increases. We also discuss
potential solutions and find bootstrapping with an intermediate model can
mitigate the deception to some extent. Our work highlights the urgent need to
pay more attention to the true reliability of superalignment.
6672b1f89cd9faec9344faf4
Wenkai Yang
claimed_verified
2024-06-19T20:18:50.819Z
64b7df742f5a966b973e25f7
/avatars/e24e7769188d441317b3b7d10ef8fd60.svg
Wenkai Yang
user
Keven16
6672b1f89cd9faec9344faf5
Shiqi Shen
6672b1f89cd9faec9344faf6
Guangyao Shen
6672b1f89cd9faec9344faf7
Zhi Gong
admin_assigned
2024-06-20T09:48:50.213Z
642100d8290342c5df87deb1
/avatars/0ba154877c28c1dea4a0db63eb68ddaa.svg
gongzhi
user
gongzhi
6672b1f89cd9faec9344faf8
Yankai Lin
admin_assigned
2024-06-20T09:48:38.751Z
657a651e1433ea7d44de6397
/avatars/ccfc76f94595a38ff4a80f77c911eabf.svg
Yankai Lin
user
lyk423
Super(ficial)-alignment: Strong Models May Deceive Weak Models in
  Weak-to-Strong Generalization
Superalignment, where humans are weak supervisors of superhuman models, has
become an important and widely discussed issue in the current era of rapid
development of Large Language Models (LLMs). The recent work preliminarily
studies this problem by using weak models to supervise strong models. It
discovers that weakly supervised strong students can consistently outperform
weak teachers towards the alignment target, leading to a weak-to-strong
generalization phenomenon. However, we are concerned that behind such a
promising phenomenon, whether there exists an issue of weak-to-strong
deception, where strong models may deceive weak models by exhibiting
well-aligned in areas known to weak models but producing misaligned behaviors
in cases weak models do not know. We then take an initial step towards
exploring this security issue in a specific but realistic multi-objective
alignment case, where there may be some alignment targets conflicting with each
other (e.g., helpfulness v.s. harmlessness). Such a conflict is likely to cause
strong models to deceive weak models in one alignment dimension to gain high
reward in other alignment dimension. Our experiments on both the reward
modeling task and the preference optimization scenario indicate: (1) the
weak-to-strong deception exists; (2) the deception phenomenon may intensify as
the capability gap between weak and strong models increases. We also discuss
potential solutions and find bootstrapping with an intermediate model can
mitigate the deception to some extent. Our work highlights the urgent need to
pay more attention to the true reliability of superalignment.
Research on superalignment reveals that weak models can outperform strong student models in alignment but may also deceive them, especially in conflicting multi-objective scenarios, necessitating careful consideration of reliability.
superalignment Large Language Models (LLMs) weak-to-strong generalization weak-to-strong deception multi-objective alignment reward modeling preference optimization bootstrapping