Debate Helps Weak-to-Strong Generalization
Common methods for aligning already-capable models with desired behavior rely
on the ability of humans to provide supervision. However, future superhuman
models will surpass the capability of humans. Therefore, humans will only be
able to weakly supervise superhuman models. This expected deficiency of human
evaluation would weaken the safety of future AI systems. Scalable oversight and
weak-to-strong generalization are two complementary approaches to tackle this
issue. In this paper, we attempt to combine the strengths of these two
approaches to further improve alignment. Specifically, we investigate ways of
improving human supervision with a strong pretrained model and then supervise
the strong model with enhanced weak human supervision. To make iterative
empirical progress, we consider an analogy: can we use a strong model to
improve weak model supervision and then use it to supervise the strong model?
We empirically test it by finetuning a small weak model on ground truth labels
with the additional help from a large strong model, and then finetuning the
strong model on labels generated by the weak model. We find that debate can
assist a weak model in extracting trustworthy information from an untrustworthy
strong model, which provides leverage as context on samples when training a
weak model. We also show that an ensemble of weak models helps exploit long
arguments generated by strong model debaters and obtain a more robust
supervision estimate. Extensive experiments on the OpenAI weak-to-strong NLP
benchmarks show that the combination approach leads to better alignment, which
indicates that debate has the potential to help weak-to-strong generalization.
6793188b56f015277a9ed95c
Hao Lang
admin_assigned
2025-01-24T10:09:24.142Z
65a6131fee7aa779f5bf8329
/avatars/aa25cc3153fd7e511b51b801e8107564.svg
langhao
user
langnick
6793188b56f015277a9ed95d
Fei Huang
admin_assigned
2025-01-24T10:09:39.353Z
635b8b6a37c6a2c12e2cce00
/avatars/229fb72180529141515d1df797b33709.svg
Fei Huang
user
hzhwcmhf
6793188b56f015277a9ed95e
Yongbin Li
admin_assigned
2025-01-24T10:08:59.394Z
66641b2fd8e1e34bc621e688
https://cdn-avatars.huggingface.co/v1/production/uploads/66641b2fd8e1e34bc621e688/csPETwnx2zCIHSWi9uAi-.png
Yongbin Li
user
Yongbin-Li
Debate Helps Weak-to-Strong Generalization
Common methods for aligning already-capable models with desired behavior rely
on the ability of humans to provide supervision. However, future superhuman
models will surpass the capability of humans. Therefore, humans will only be
able to weakly supervise superhuman models. This expected deficiency of human
evaluation would weaken the safety of future AI systems. Scalable oversight and
weak-to-strong generalization are two complementary approaches to tackle this
issue. In this paper, we attempt to combine the strengths of these two
approaches to further improve alignment. Specifically, we investigate ways of
improving human supervision with a strong pretrained model and then supervise
the strong model with enhanced weak human supervision. To make iterative
empirical progress, we consider an analogy: can we use a strong model to
improve weak model supervision and then use it to supervise the strong model?
We empirically test it by finetuning a small weak model on ground truth labels
with the additional help from a large strong model, and then finetuning the
strong model on labels generated by the weak model. We find that debate can
assist a weak model in extracting trustworthy information from an untrustworthy
strong model, which provides leverage as context on samples when training a
weak model. We also show that an ensemble of weak models helps exploit long
arguments generated by strong model debaters and obtain a more robust
supervision estimate. Extensive experiments on the OpenAI weak-to-strong NLP
benchmarks show that the combination approach leads to better alignment, which
indicates that debate has the potential to help weak-to-strong generalization.
Debate between weak and strong models improves the alignment of AI systems by extracting trustworthy information and providing robust supervision estimates.
weak supervision superhuman models scalable oversight weak-to-strong generalization fine-tuning debate ensemble methods OpenAI weak-to-strong NLP benchmarks