Suppressing Pink Elephants with Direct Principle Feedback
Existing methods for controlling language models, such as RLHF and
Constitutional AI, involve determining which LLM behaviors are desirable and
training them into a language model. However, in many cases, it is desirable
for LLMs to be controllable at inference time, so that they can be
used in multiple contexts with diverse needs. We illustrate this with the
Pink Elephant Problem: instructing an LLM to avoid discussing a
certain entity (a ``Pink Elephant''), and instead discuss a preferred entity
(``Grey Elephant''). We apply a novel simplification of Constitutional AI,
Direct Principle Feedback, which skips the ranking of responses and
uses DPO directly on critiques and revisions. Our results show that after DPF
fine-tuning on our synthetic Pink Elephants dataset, our 13B fine-tuned LLaMA 2
model significantly outperforms Llama-2-13B-Chat and a prompted baseline, and
performs as well as GPT-4 in on our curated test set assessing the Pink
Elephant Problem.
65cad2814913057ac8189f99
Louis Castricato
admin_assigned
2024-02-13T12:20:26.331Z
60d35a0307da9c17c7270909
https://cdn-avatars.huggingface.co/v1/production/uploads/1665155571839-60d35a0307da9c17c7270909.jpeg
Louis Castricato
user
LouisCastricato
65cad2814913057ac8189f9a
Nathan Lile
admin_assigned
2024-03-25T12:35:39.367Z
61aa15fd8a9625ebfe284286
https://cdn-avatars.huggingface.co/v1/production/uploads/61aa15fd8a9625ebfe284286/KaGzIeijcgcN15JErCqft.jpeg
nathan lile
user
nlile
65cad2814913057ac8189f9b
Suraj Anand
admin_assigned
2024-02-13T12:21:10.203Z
65a8015fc0e51d82e67e9167
/avatars/e13f5bc548dbe129694e4d3536e8057a.svg
Suraj Anand
user
surajk610
65cad2814913057ac8189f9c
Hailey Schoelkopf
admin_assigned
2024-02-13T12:21:17.178Z
62895a0215aeee85756062c4
https://cdn-avatars.huggingface.co/v1/production/uploads/1669665010552-62895a0215aeee85756062c4.jpeg
Hailey Schoelkopf
user
hails
65cad2814913057ac8189f9d
Siddharth Verma
65cad2814913057ac8189f9e
Stella Biderman
claimed_verified
2024-02-13T08:10:12.517Z
60347d3660e3dd96631c9093
https://cdn-avatars.huggingface.co/v1/production/uploads/60347d3660e3dd96631c9093/B3fuZer5N04tZIAYrLnz4.jpeg
Stella Biderman
user
stellaathena
Suppressing Pink Elephants with Direct Principle Feedback
Existing methods for controlling language models, such as RLHF and
Constitutional AI, involve determining which LLM behaviors are desirable and
training them into a language model. However, in many cases, it is desirable
for LLMs to be controllable at inference time, so that they can be
used in multiple contexts with diverse needs. We illustrate this with the
Pink Elephant Problem: instructing an LLM to avoid discussing a
certain entity (a ``Pink Elephant''), and instead discuss a preferred entity
(``Grey Elephant''). We apply a novel simplification of Constitutional AI,
Direct Principle Feedback, which skips the ranking of responses and
uses DPO directly on critiques and revisions. Our results show that after DPF
fine-tuning on our synthetic Pink Elephants dataset, our 13B fine-tuned LLaMA 2
model significantly outperforms Llama-2-13B-Chat and a prompted baseline, and
performs as well as GPT-4 in on our curated test set assessing the Pink
Elephant Problem.
A novel method called Direct Principle Feedback fine-tunes language models to become controllable at inference time, demonstrated on the Pink Elephant Problem.
RLHF Constitutional AI Direct Principle Feedback DPO Pink Elephant Problem fine-tuning LLaMA 2 GPT-4