Improving Open Language Models by Learning from Organic Interactions
We present BlenderBot 3x, an update on the conversational model BlenderBot 3,
which is now trained using organic conversation and feedback data from
participating users of the system in order to improve both its skills and
safety. We are publicly releasing the participating de-identified interaction
data for use by the research community, in order to spur further progress.
Training models with organic data is challenging because interactions with
people "in the wild" include both high quality conversations and feedback, as
well as adversarial and toxic behavior. We study techniques that enable
learning from helpful teachers while avoiding learning from people who are
trying to trick the model into unhelpful or toxic responses. BlenderBot 3x is
both preferred in conversation to BlenderBot 3, and is shown to produce safer
responses in challenging situations. While our current models are still far
from perfect, we believe further improvement can be achieved by continued use
of the techniques explored in this work.
64828cc35f581f451849684c
Jing Xu
64828cc35f581f451849684d
Da Ju
64828cc35f581f451849684e
Joshua Lane
64828cc35f581f451849684f
Mojtaba Komeili
claimed_verified
2024-06-27T09:48:52.176Z
659d717f2dbf5c8fee9c1fac
/avatars/6bea1195499a66da9b48fe0c27781ba8.svg
Mojtaba Komeili
user
mkomeili
64828cc35f581f4518496850
Eric Michael Smith
64828cc35f581f4518496851
Megan Ung
admin_assigned
2023-06-09T16:53:55.559Z
63c6f88850cc81901da905f3
/avatars/e75bb87af183aa5c3c5a7d09f2de1d60.svg
Megan Ung
user
meganung
64828cc35f581f4518496852
Morteza Behrooz
admin_assigned
2023-06-12T13:46:40.806Z
64597ca038d95f3427a478e2
/avatars/124de43a636728193e549460aa0f800c.svg
Morteza Behrooz
user
lessteza
64828cc35f581f4518496853
William Ngan
admin_assigned
2023-06-12T13:47:00.053Z
630414995c70c21d0ead52b0
/avatars/33cd9046d9833f0a9bc945ae776778e6.svg
William Ngan
user
metaphorical
64828cc35f581f4518496854
Rashel Moritz
64828cc35f581f4518496855
Sainbayar Sukhbaatar
claimed_verified
2024-07-30T16:01:37.228Z
66a8611eb51510d82ed54231
/avatars/ad559e774fee4914091b82c9831ae2a2.svg
Sainbayar Sukhbaatar
user
sainbar
64828cc35f581f4518496856
Y-Lan Boureau
claimed_verified
2023-07-10T09:25:19.383Z
64a82a24a1cd4978d49aa7cb
/avatars/d7d7c3ad870541017551594bd30b6fde.svg
Y-Lan Boureau
user
notebleue
64828cc35f581f4518496857
Jason Weston
64828cc35f581f4518496858
Kurt Shuster
Improving Open Language Models by Learning from Organic Interactions
We present BlenderBot 3x, an update on the conversational model BlenderBot 3,
which is now trained using organic conversation and feedback data from
participating users of the system in order to improve both its skills and
safety. We are publicly releasing the participating de-identified interaction
data for use by the research community, in order to spur further progress.
Training models with organic data is challenging because interactions with
people "in the wild" include both high quality conversations and feedback, as
well as adversarial and toxic behavior. We study techniques that enable
learning from helpful teachers while avoiding learning from people who are
trying to trick the model into unhelpful or toxic responses. BlenderBot 3x is
both preferred in conversation to BlenderBot 3, and is shown to produce safer
responses in challenging situations. While our current models are still far
from perfect, we believe further improvement can be achieved by continued use
of the techniques explored in this work.
BlenderBot 3x, an updated conversational model, is trained using organic user data to enhance its skills and safety, showing preference in conversations and better handling of challenging situations.
conversational model organic conversation feedback data training adversarial behavior toxic behavior helpful teachers learning techniques safety challenging situations