Skills-in-Context Prompting: Unlocking Compositionality in Large
  Language Models
We consider the problem of eliciting compositional generalization
capabilities in large language models (LLMs) with a novel type of prompting
strategy. Compositional generalization empowers the LLMs to solve problems that
are harder than the ones they have seen (i.e., easy-to-hard generalization),
which is a critical reasoning capability of human-like intelligence. However,
even the current state-of-the-art LLMs still struggle with this form of
reasoning. To bridge this gap, we propose skills-in-context (SKiC) prompting,
which instructs LLMs how to compose basic skills to resolve more complex
problems. We find that it is crucial to demonstrate both the skills and the
compositional examples within the same prompting context. With as few as two
examplars, our SKiC prompting initiates strong synergies between skills and
their composition capabilities. Notably, it empowers LLMs to solve unseen
problems that require innovative skill compositions, achieving near-perfect
generalization on a broad range of challenging compositionality tasks.
Intriguingly, SKiC prompting unlocks the latent potential of LLMs, enabling
them to leverage pre-existing internal skills acquired during earlier
pre-training stages, even when these skills are not explicitly presented in the
prompting context. This results in the capability of LLMs to solve unseen
complex problems by activating and composing internal competencies. With such
prominent features, SKiC prompting is able to achieve state-of-the-art
performance on challenging mathematical reasoning benchmarks (e.g., MATH).
64c9b91af7f4ccb5ea6573de
Jiaao Chen
admin_assigned
2023-08-02T07:45:41.838Z
6327c45946ffcfc27cfa1ee3
/avatars/f64b36e978fc369bfdfebc16057cd67e.svg
Chen
user
Jiaao
64c9b91af7f4ccb5ea6573df
Xiaoman Pan
64c9b91af7f4ccb5ea6573e0
Dian Yu
claimed_verified
2024-06-20T07:35:16.524Z
62d58fd53bf5e059f7cc3245
/avatars/7a4f3ee4a37245f67efd26749d66a706.svg
Dian Yu
user
yudian
64c9b91af7f4ccb5ea6573e1
Kaiqiang Song
claimed_verified
2023-08-07T12:38:53.463Z
63a261f13c003e409322c191
/avatars/dc4be6ce1983d0d32620275df124fe45.svg
Kaiqiang Song
user
kqsong
64c9b91af7f4ccb5ea6573e2
Xiaoyang Wang
claimed_verified
2023-12-18T08:48:52.155Z
657cd228138b7e391444a65d
/avatars/c7c984ae483144fab627aa2c54d91d0f.svg
Xiaoyang Wang
user
xywang1
64c9b91af7f4ccb5ea6573e3
Dong Yu
64c9b91af7f4ccb5ea6573e4
Jianshu Chen
admin_assigned
2023-08-02T07:48:36.355Z
648353887641590704d53894
/avatars/d0968259531561f8c93f37bddefe2268.svg
Jianshu Chen
user
chenjianshu
Skills-in-Context Prompting: Unlocking Compositionality in Large
  Language Models
We consider the problem of eliciting compositional generalization
capabilities in large language models (LLMs) with a novel type of prompting
strategy. Compositional generalization empowers the LLMs to solve problems that
are harder than the ones they have seen (i.e., easy-to-hard generalization),
which is a critical reasoning capability of human-like intelligence. However,
even the current state-of-the-art LLMs still struggle with this form of
reasoning. To bridge this gap, we propose skills-in-context (SKiC) prompting,
which instructs LLMs how to compose basic skills to resolve more complex
problems. We find that it is crucial to demonstrate both the skills and the
compositional examples within the same prompting context. With as few as two
examplars, our SKiC prompting initiates strong synergies between skills and
their composition capabilities. Notably, it empowers LLMs to solve unseen
problems that require innovative skill compositions, achieving near-perfect
generalization on a broad range of challenging compositionality tasks.
Intriguingly, SKiC prompting unlocks the latent potential of LLMs, enabling
them to leverage pre-existing internal skills acquired during earlier
pre-training stages, even when these skills are not explicitly presented in the
prompting context. This results in the capability of LLMs to solve unseen
complex problems by activating and composing internal competencies. With such
prominent features, SKiC prompting is able to achieve state-of-the-art
performance on challenging mathematical reasoning benchmarks (e.g., MATH).
A novel prompting strategy called skills-in-context (SKiC) enhances compositional generalization in large language models by enabling them to solve unseen complex problems through the synergistic activation and combination of internal skills.
compositional generalization large language models LLMs skills-in-context SKiC prompting generalization reasoning benchmarks MATH