Teaching Language Models to Self-Improve through Interactive
  Demonstrations
The self-improving ability of large language models (LLMs), enabled by
prompting them to analyze and revise their own outputs, has garnered
significant interest in recent research. However, this ability has been shown
to be absent and difficult to learn for smaller models, thus widening the
performance gap between state-of-the-art LLMs and more cost-effective and
faster ones. To reduce this gap, we introduce TriPosT, a training algorithm
that endows smaller models with such self-improvement ability, and show that
our approach can improve a LLaMA-7b's performance on math and reasoning tasks
by up to 7.13%. In contrast to prior work, we achieve this by using the smaller
model to interact with LLMs to collect feedback and improvements on its own
generations. We then replay this experience to train the small model. Our
experiments on four math and reasoning datasets show that the interactive
experience of learning from and correcting its own mistakes is crucial for
small models to improve their performance.
6535eeb8e983fb23fa695eda
Xiao Yu
claimed_verified
2023-10-23T14:46:35.296Z
6234fd736dcfc5fe9f5b8601
https://cdn-avatars.huggingface.co/v1/production/uploads/1647639915850-noauth.jpeg
Xiao Yu
user
jasonyux
6535eeb8e983fb23fa695edb
Baolin Peng
admin_assigned
2023-10-23T07:40:35.014Z
61942296d5c2ba6daa290357
/avatars/594021cc183c4922d48b46f43772a062.svg
Baolin Peng
user
Baolin
6535eeb8e983fb23fa695edc
Michel Galley
admin_assigned
2023-10-23T07:40:47.113Z
5f6bed855e78cc6b0ed31dee
/avatars/d67edb079fa3fe7ea6f2081f7a3afe9e.svg
Michel Galley
user
mgalley
6535eeb8e983fb23fa695edd
Jianfeng Gao
admin_assigned
2023-10-23T07:40:54.789Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
6535eeb8e983fb23fa695ede
Zhou Yu
Teaching Language Models to Self-Improve through Interactive
  Demonstrations
The self-improving ability of large language models (LLMs), enabled by
prompting them to analyze and revise their own outputs, has garnered
significant interest in recent research. However, this ability has been shown
to be absent and difficult to learn for smaller models, thus widening the
performance gap between state-of-the-art LLMs and more cost-effective and
faster ones. To reduce this gap, we introduce TriPosT, a training algorithm
that endows smaller models with such self-improvement ability, and show that
our approach can improve a LLaMA-7b's performance on math and reasoning tasks
by up to 7.13%. In contrast to prior work, we achieve this by using the smaller
model to interact with LLMs to collect feedback and improvements on its own
generations. We then replay this experience to train the small model. Our
experiments on four math and reasoning datasets show that the interactive
experience of learning from and correcting its own mistakes is crucial for
small models to improve their performance.
TriPosT trains smaller models to self-improve by interacting with larger language models to collect feedback, narrowing the performance gap in math and reasoning tasks.
large language models self-improving ability prompting smaller models TriPosT training algorithm LLaMA-7b math tasks reasoning tasks interactive experience datasets