In-Context Principle Learning from Mistakes
In-context learning (ICL, also known as few-shot prompting) has been the
standard method of adapting LLMs to downstream tasks, by learning from a few
input-output examples. Nonetheless, all ICL-based approaches only learn from
correct input-output pairs. In this paper, we revisit this paradigm, by
learning more from the few given input-output examples. We introduce Learning
Principles (LEAP): First, we intentionally induce the model to make mistakes on
these few examples; then we reflect on these mistakes, and learn explicit
task-specific "principles" from them, which help solve similar problems and
avoid common mistakes; finally, we prompt the model to answer unseen test
questions using the original few-shot examples and these learned general
principles. We evaluate LEAP on a wide range of benchmarks, including multi-hop
question answering (Hotpot QA), textual QA (DROP), Big-Bench Hard reasoning,
and math problems (GSM8K and MATH); in all these benchmarks, LEAP improves the
strongest available LLMs such as GPT-3.5-turbo, GPT-4, GPT-4 turbo and
Claude-2.1. For example, LEAP improves over the standard few-shot prompting
using GPT-4 by 7.5% in DROP, and by 3.3% in HotpotQA. Importantly, LEAP does
not require any more input or examples than the standard few-shot prompting
settings.
65c5915184ed215272356bcd
Tianjun Zhang
extracted_confirmed
2024-08-01T05:49:17.162Z
6374cd6b6ea8da14f8fef8dc
https://cdn-avatars.huggingface.co/v1/production/uploads/6374cd6b6ea8da14f8fef8dc/l13bg0tKDjCnUw3I895QZ.png
Tianjun Zhang
user
tianjunz
65c5915184ed215272356bce
Aman Madaan
admin_assigned
2024-02-09T10:32:23.977Z
61bbc2b2d96f5d4111901dcb
/avatars/83c826d5cad7bab27d33585cb7e64575.svg
Aman Madaan
user
amanmadaan
65c5915184ed215272356bcf
Luyu Gao
admin_assigned
2024-02-09T10:32:30.523Z
6009aa9126e23ab5edb47afd
/avatars/f8afa9cf648227ad5817323451ede378.svg
Luyu Gao
user
Luyu
65c5915184ed215272356bd0
Steven Zheng
65c5915184ed215272356bd1
Swaroop Mishra
admin_assigned
2024-02-09T10:32:56.676Z
61841feb7c380c66d40cad54
/avatars/7a79362a24b844860781fb0625527eae.svg
Swaroop Mishra
user
swaroop7
65c5915184ed215272356bd2
Yiming Yang
65c5915184ed215272356bd3
Niket Tandon
admin_assigned
2024-02-09T10:33:20.842Z
62e69a709bb6d949274ad32b
/avatars/b6a796d12a1c1c708f5b46c9082cdee6.svg
Niket Tandon
user
nikett
65c5915184ed215272356bd4
Uri Alon
admin_assigned
2024-02-09T10:33:39.720Z
62cd8153248f9e6bc20ab250
https://cdn-avatars.huggingface.co/v1/production/uploads/1657635637504-62cd8153248f9e6bc20ab250.jpeg
Uri Alon
user
urialon
In-Context Principle Learning from Mistakes
In-context learning (ICL, also known as few-shot prompting) has been the
standard method of adapting LLMs to downstream tasks, by learning from a few
input-output examples. Nonetheless, all ICL-based approaches only learn from
correct input-output pairs. In this paper, we revisit this paradigm, by
learning more from the few given input-output examples. We introduce Learning
Principles (LEAP): First, we intentionally induce the model to make mistakes on
these few examples; then we reflect on these mistakes, and learn explicit
task-specific "principles" from them, which help solve similar problems and
avoid common mistakes; finally, we prompt the model to answer unseen test
questions using the original few-shot examples and these learned general
principles. We evaluate LEAP on a wide range of benchmarks, including multi-hop
question answering (Hotpot QA), textual QA (DROP), Big-Bench Hard reasoning,
and math problems (GSM8K and MATH); in all these benchmarks, LEAP improves the
strongest available LLMs such as GPT-3.5-turbo, GPT-4, GPT-4 turbo and
Claude-2.1. For example, LEAP improves over the standard few-shot prompting
using GPT-4 by 7.5% in DROP, and by 3.3% in HotpotQA. Importantly, LEAP does
not require any more input or examples than the standard few-shot prompting
settings.
LEAP, a novel few-shot learning approach, enhances LLM performance by inducing and reflecting on mistakes from initial examples to learn task-specific principles, applied across various benchmarks.
in-context learning few-shot prompting few-shot examples Learning Principles task-specific principles multi-hop question answering textual QA Big-Bench Hard reasoning math problems GPT-3.5-turbo GPT-4 GPT-4 turbo Claude-2.1 DROP HotpotQA