Understanding In-Context Learning via Supportive Pretraining Data
In-context learning (ICL) improves language models' performance on a variety
of NLP tasks by simply demonstrating a handful of examples at inference time.
It is not well understood why ICL ability emerges, as the model has never been
specifically trained on such demonstrations. Unlike prior work that explores
implicit mechanisms behind ICL, we study ICL via investigating the pretraining
data. Specifically, we first adapt an iterative, gradient-based approach to
find a small subset of pretraining data that supports ICL. We observe that a
continued pretraining on this small subset significantly improves the model's
ICL ability, by up to 18%. We then compare the supportive subset constrastively
with random subsets of pretraining data and discover: (1) The supportive
pretraining data to ICL do not have a higher domain relevance to downstream
tasks. (2) The supportive pretraining data have a higher mass of rarely
occurring, long-tail tokens. (3) The supportive pretraining data are
challenging examples where the information gain from long-range context is
below average, indicating learning to incorporate difficult long-range context
encourages ICL. Our work takes a first step towards understanding ICL via
analyzing instance-level pretraining data. Our insights have a potential to
enhance the ICL ability of language models by actively guiding the construction
of pretraining data in the future.
649b9396b38579156347eca6
Xiaochuang Han
claimed_verified
2023-06-29T20:55:52.123Z
635ea3969f24f6db0a1e2d0b
https://cdn-avatars.huggingface.co/v1/production/uploads/635ea3969f24f6db0a1e2d0b/RxaJKAx4MYMShb58yuUxy.png
Xiaochuang Han
user
xhan77
649b9396b38579156347eca7
Daniel Simig
extracted_pending
2023-08-24T05:37:15.784Z
64e6ea7a3b9c1e300efb79ca
/avatars/393898e6437d82dd143e137d6f9d1647.svg
Daniel Simig
user
danielsimig
649b9396b38579156347eca8
Todor Mihaylov
649b9396b38579156347eca9
Yulia Tsvetkov
649b9396b38579156347ecaa
Asli Celikyilmaz
649b9396b38579156347ecab
Tianlu Wang
claimed_verified
2023-06-28T20:15:50.503Z
63f6a0e6b4c9a104f4b95d86
/avatars/660c3e44bea976f3e0560bf95466b03c.svg
Tianlu Wang
user
Tianlu
Understanding In-Context Learning via Supportive Pretraining Data
In-context learning (ICL) improves language models' performance on a variety
of NLP tasks by simply demonstrating a handful of examples at inference time.
It is not well understood why ICL ability emerges, as the model has never been
specifically trained on such demonstrations. Unlike prior work that explores
implicit mechanisms behind ICL, we study ICL via investigating the pretraining
data. Specifically, we first adapt an iterative, gradient-based approach to
find a small subset of pretraining data that supports ICL. We observe that a
continued pretraining on this small subset significantly improves the model's
ICL ability, by up to 18%. We then compare the supportive subset constrastively
with random subsets of pretraining data and discover: (1) The supportive
pretraining data to ICL do not have a higher domain relevance to downstream
tasks. (2) The supportive pretraining data have a higher mass of rarely
occurring, long-tail tokens. (3) The supportive pretraining data are
challenging examples where the information gain from long-range context is
below average, indicating learning to incorporate difficult long-range context
encourages ICL. Our work takes a first step towards understanding ICL via
analyzing instance-level pretraining data. Our insights have a potential to
enhance the ICL ability of language models by actively guiding the construction
of pretraining data in the future.
Investigating the pretraining data reveals that long-tail tokens and challenging examples with low long-range context contribute to the in-context learning ability of language models.
in-context learning ICL gradient-based approach long-tail tokens long-range context