Reasons to Reject? Aligning Language Models with Judgments
As humans, we consistently engage in interactions with our peers and receive
feedback in the form of natural language. This language feedback allows us to
reflect on our actions, maintain appropriate behavior, and rectify our errors.
The question arises naturally: can we use language feedback to align large
language models (LLMs)? In contrast to previous research that aligns LLMs with
reward or preference data, we present the first systematic exploration of
alignment through the lens of language feedback (i.e., judgment). We commence
with an in-depth investigation of potential methods that can be adapted for
aligning LLMs with judgments, revealing that these methods are unable to fully
capitalize on the judgments. To facilitate more effective utilization of
judgments, we propose a novel framework, Contrastive Unlikelihood Training
(CUT), that allows for fine-grained inappropriate content detection and
correction based on judgments. Our offline alignment results show that, with
merely 1317 off-the-shelf judgment data, CUT (LLaMA2-13b) can beat the 175B
DaVinci003 and surpass the best baseline by 52.34 points on AlpacaEval. The
online alignment results demonstrate that CUT can align LLMs (LLaMA2-chat-13b)
in an iterative fashion using model-specific judgment data, with a steady
performance improvement from 81.09 to 91.36 points on AlpacaEval. Our analysis
further suggests that judgments exhibit greater potential than rewards for LLM
alignment and warrant future research.
658a49e94f349f95cffb27d3
Weiwen Xu
claimed_verified
2023-12-26T10:36:59.141Z
64118689756b9e455c7eac62
/avatars/cdb3da22593facf545a0bafbf548b07e.svg
Xu Weiwen
user
xww033
658a49e94f349f95cffb27d4
Deng Cai
658a49e94f349f95cffb27d5
Zhisong Zhang
658a49e94f349f95cffb27d6
Wai Lam
658a49e94f349f95cffb27d7
Shuming Shi
Reasons to Reject? Aligning Language Models with Judgments
As humans, we consistently engage in interactions with our peers and receive
feedback in the form of natural language. This language feedback allows us to
reflect on our actions, maintain appropriate behavior, and rectify our errors.
The question arises naturally: can we use language feedback to align large
language models (LLMs)? In contrast to previous research that aligns LLMs with
reward or preference data, we present the first systematic exploration of
alignment through the lens of language feedback (i.e., judgment). We commence
with an in-depth investigation of potential methods that can be adapted for
aligning LLMs with judgments, revealing that these methods are unable to fully
capitalize on the judgments. To facilitate more effective utilization of
judgments, we propose a novel framework, Contrastive Unlikelihood Training
(CUT), that allows for fine-grained inappropriate content detection and
correction based on judgments. Our offline alignment results show that, with
merely 1317 off-the-shelf judgment data, CUT (LLaMA2-13b) can beat the 175B
DaVinci003 and surpass the best baseline by 52.34 points on AlpacaEval. The
online alignment results demonstrate that CUT can align LLMs (LLaMA2-chat-13b)
in an iterative fashion using model-specific judgment data, with a steady
performance improvement from 81.09 to 91.36 points on AlpacaEval. Our analysis
further suggests that judgments exhibit greater potential than rewards for LLM
alignment and warrant future research.
A novel framework, Contrastive Unlikelihood Training (CUT), effectively aligns large language models using language feedback, outperforming existing methods with less data and iterative improvements.
large language models language feedback judgment Contrastive Unlikelihood Training inappropriate content detection correction offline alignment online alignment AlpacaEval reward LLaMA2-13b DaVinci003