Prompt Engineering a Prompt Engineer
Prompt engineering is a challenging yet crucial task for optimizing the
performance of large language models (LLMs). It requires complex reasoning to
examine the model's errors, hypothesize what is missing or misleading in the
current prompt, and communicate the task with clarity. While recent works
indicate that LLMs can be meta-prompted to perform automatic prompt
engineering, their potentials may not be fully untapped due to the lack of
sufficient guidance to elicit complex reasoning capabilities in LLMs in the
meta-prompt. In this work, we investigate the problem of "prompt engineering a
prompt engineer" -- constructing a meta-prompt that more effectively guides
LLMs to perform automatic prompt engineering. We introduce and analyze key
components, such as a step-by-step reasoning template and context
specification, which lead to improved performance. In addition, inspired by
common optimization concepts such as batch size, step size and momentum, we
introduce their verbalized counterparts to the meta-prompt and investigate
their effects. Our final method, named PE2, finds a prompt that outperforms
"let's think step by step" by 6.3% on the MultiArith dataset and 3.1% on the
GSM8K dataset. To demonstrate its versatility, we apply PE2 to the Instruction
Induction benchmark, a suite of counterfactual tasks, and a lengthy, real-world
industrial prompt. In these settings, PE2 achieves strong performance and
outperforms prior automatic prompt engineering baselines. Further, we show that
PE2 makes meaningful and targeted prompt edits, amends erroneous or incomplete
prompts, and presents non-trivial counterfactual reasoning abilities.
65519961685ba4c13dae2616
Qinyuan Ye
extracted_confirmed
2024-01-11T22:13:36.656Z
64a3a76868fe7eed8fc1a72c
https://cdn-avatars.huggingface.co/v1/production/uploads/64a3a76868fe7eed8fc1a72c/1aNwBBt395r7PHB5yCiW2.jpeg
Qinyuan Ye
user
qinyuany
65519961685ba4c13dae2617
Maxamed Axmed
65519961685ba4c13dae2618
Reid Pryzant
admin_assigned
2023-11-13T10:44:40.445Z
64e3d38c618cd90997ee0138
/avatars/847210a609b156fdb9d824cc815bde27.svg
Reid Pryzant
user
rpryzant
65519961685ba4c13dae2619
Fereshte Khani
Prompt Engineering a Prompt Engineer
Prompt engineering is a challenging yet crucial task for optimizing the
performance of large language models (LLMs). It requires complex reasoning to
examine the model's errors, hypothesize what is missing or misleading in the
current prompt, and communicate the task with clarity. While recent works
indicate that LLMs can be meta-prompted to perform automatic prompt
engineering, their potentials may not be fully untapped due to the lack of
sufficient guidance to elicit complex reasoning capabilities in LLMs in the
meta-prompt. In this work, we investigate the problem of "prompt engineering a
prompt engineer" -- constructing a meta-prompt that more effectively guides
LLMs to perform automatic prompt engineering. We introduce and analyze key
components, such as a step-by-step reasoning template and context
specification, which lead to improved performance. In addition, inspired by
common optimization concepts such as batch size, step size and momentum, we
introduce their verbalized counterparts to the meta-prompt and investigate
their effects. Our final method, named PE2, finds a prompt that outperforms
"let's think step by step" by 6.3% on the MultiArith dataset and 3.1% on the
GSM8K dataset. To demonstrate its versatility, we apply PE2 to the Instruction
Induction benchmark, a suite of counterfactual tasks, and a lengthy, real-world
industrial prompt. In these settings, PE2 achieves strong performance and
outperforms prior automatic prompt engineering baselines. Further, we show that
PE2 makes meaningful and targeted prompt edits, amends erroneous or incomplete
prompts, and presents non-trivial counterfactual reasoning abilities.
The study develops a meta-prompting technique, PE2, to enhance automatic prompt engineering for large language models, demonstrating improved performance and sophisticated reasoning capabilities across various datasets and tasks.
large language models prompt engineering meta-prompting step-by-step reasoning template context specification batch size step size momentum MultiArith dataset GSM8K dataset Instruction Induction benchmark counterfactual tasks industrial prompt