Tailoring Self-Rationalizers with Multi-Reward Distillation
Large language models (LMs) are capable of generating free-text rationales to
aid question answering. However, prior work 1) suggests that useful
self-rationalization is emergent only at significant scales (e.g., 175B
parameter GPT-3); and 2) focuses largely on downstream performance, ignoring
the semantics of the rationales themselves, e.g., are they faithful, true, and
helpful for humans? In this work, we enable small-scale LMs (approx. 200x
smaller than GPT-3) to generate rationales that not only improve downstream
task performance, but are also more plausible, consistent, and diverse,
assessed both by automatic and human evaluation. Our method, MaRio
(Multi-rewArd RatIOnalization), is a multi-reward conditioned
self-rationalization algorithm that optimizes multiple distinct properties like
plausibility, diversity and consistency. Results on five difficult
question-answering datasets StrategyQA, QuaRel, OpenBookQA, NumerSense and QASC
show that not only does MaRio improve task accuracy, but it also improves the
self-rationalization quality of small LMs across the aforementioned axes better
than a supervised fine-tuning (SFT) baseline. Extensive human evaluations
confirm that MaRio rationales are preferred vs. SFT rationales, as well as
qualitative improvements in plausibility and consistency.
6549c6d51718116e3508e005
Sahana Ramnath
extracted_pending
2023-12-04T07:31:46.669Z
656d803c6f39f1565858aa9c
/avatars/d6e30599cf32bd13e0b96a85cff3f9ad.svg
Sahana Ramnath
user
SahanaRamnath
6549c6d51718116e3508e006
Brihi Joshi
admin_assigned
2023-11-07T17:19:37.667Z
64ee4d611c9470d52f5d8aaf
/avatars/223617a9933c833e7ea424bfcc3883c0.svg
Brihi Joshi
user
brihijoshi
6549c6d51718116e3508e007
Skyler Hallinan
admin_assigned
2023-11-07T17:19:44.231Z
637d38fcb8e573d75bedc033
https://cdn-avatars.huggingface.co/v1/production/uploads/637d38fcb8e573d75bedc033/MbzfbMBGN2ajH7oZtitcB.png
Skyler Hallinan
user
hallisky
6549c6d51718116e3508e008
Ximing Lu
admin_assigned
2023-11-07T17:19:50.642Z
640928bd3461c51cf7378707
/avatars/b29fcb7388b81f8686086352f6321d06.svg
Ximing Lu
user
Ximing
6549c6d51718116e3508e009
Liunian Harold Li
admin_assigned
2023-11-07T17:19:57.434Z
60b7c68c105296d03a0530ca
/avatars/e5823c7a8f7b77b4d75125192c1adc66.svg
Liunian Harold Li
user
harold
6549c6d51718116e3508e00a
Aaron Chan
admin_assigned
2023-11-07T17:20:05.715Z
6426440722bddcea3d27ee8a
/avatars/daac33a79a6b35c36154274b6f109f8c.svg
Aaron_Chan
user
AaronChan
6549c6d51718116e3508e00b
Jack Hessel
admin_assigned
2023-11-07T17:20:13.003Z
625498644f4edf771516b2cb
https://cdn-avatars.huggingface.co/v1/production/uploads/1664497251391-625498644f4edf771516b2cb.jpeg
Jack Hessel
user
jmhessel
6549c6d51718116e3508e00c
Yejin Choi
admin_assigned
2023-11-07T17:20:20.082Z
64d42729f63b01b7f676b176
/avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
Yejin Choi
user
yejinchoinka
6549c6d51718116e3508e00d
Xiang Ren
Tailoring Self-Rationalizers with Multi-Reward Distillation
Large language models (LMs) are capable of generating free-text rationales to
aid question answering. However, prior work 1) suggests that useful
self-rationalization is emergent only at significant scales (e.g., 175B
parameter GPT-3); and 2) focuses largely on downstream performance, ignoring
the semantics of the rationales themselves, e.g., are they faithful, true, and
helpful for humans? In this work, we enable small-scale LMs (approx. 200x
smaller than GPT-3) to generate rationales that not only improve downstream
task performance, but are also more plausible, consistent, and diverse,
assessed both by automatic and human evaluation. Our method, MaRio
(Multi-rewArd RatIOnalization), is a multi-reward conditioned
self-rationalization algorithm that optimizes multiple distinct properties like
plausibility, diversity and consistency. Results on five difficult
question-answering datasets StrategyQA, QuaRel, OpenBookQA, NumerSense and QASC
show that not only does MaRio improve task accuracy, but it also improves the
self-rationalization quality of small LMs across the aforementioned axes better
than a supervised fine-tuning (SFT) baseline. Extensive human evaluations
confirm that MaRio rationales are preferred vs. SFT rationales, as well as
qualitative improvements in plausibility and consistency.
MaRio, a multi-reward conditioned self-rationalization algorithm, enhances the quality of rationales generated by small language models, improving both performance and the semantic properties of the rationales.
large language models self-rationalization GPT-3 small-scale LMs multi-reward conditioned plausibility diversity consistency supervised fine-tuning StrategyQA QuaRel OpenBookQA NumerSense QASC