Improving Language Plasticity via Pretraining with Active Forgetting
Pretrained language models (PLMs) are today the primary model for natural
language processing. Despite their impressive downstream performance, it can be
difficult to apply PLMs to new languages, a barrier to making their
capabilities universally accessible. While prior work has shown it possible to
address this issue by learning a new embedding layer for the new language,
doing so is both data and compute inefficient. We propose to use an active
forgetting mechanism during pretraining, as a simple way of creating PLMs that
can quickly adapt to new languages. Concretely, by resetting the embedding
layer every K updates during pretraining, we encourage the PLM to improve its
ability of learning new embeddings within a limited number of updates, similar
to a meta-learning effect. Experiments with RoBERTa show that models pretrained
with our forgetting mechanism not only demonstrate faster convergence during
language adaptation but also outperform standard ones in a low-data regime,
particularly for languages that are distant from English.
64a387a702617d262f8a7cff
Yihong Chen
claimed_verified
2024-06-24T07:37:13.413Z
6674243633e45ebfcdc8be3b
/avatars/4e87009e3576c9351d70bc8223538b9e.svg
yihong
user
yihong-chen
64a387a702617d262f8a7d00
Kelly Marchisio
extracted_confirmed
2025-05-15T16:31:48.310Z
66356b1e2fcf64c1548a1233
/avatars/9a1fab7da6cd16e8bf2d104c399ae34f.svg
Kelly Marchisio
user
kellycohere
64a387a702617d262f8a7d01
Roberta Raileanu
extracted_pending
2023-07-04T02:44:56.019Z
633e94793a17ab61de8e2b9c
/avatars/5f2f58ddeed211393660ada6b135f0d5.svg
Roberta Raileanu
user
rraileanu
64a387a702617d262f8a7d02
David Ifeoluwa Adelani
claimed_verified
2023-07-04T19:16:57.055Z
5fcc1929563427b03e9af259
https://cdn-avatars.huggingface.co/v1/production/uploads/1620642852901-5fcc1929563427b03e9af259.jpeg
David Adelani
user
Davlan
64a387a702617d262f8a7d03
Pontus Stenetor
64a387a702617d262f8a7d04
Sebastian Riedel
64a387a702617d262f8a7d05
Mikel Artetx
extracted_pending
2023-09-18T08:38:05.072Z
65080c544a8839a8bda2cfbd
/avatars/8684a50e82bd21e14c9f32c3a443760d.svg
Mikel Artetxe
user
artetxem
Improving Language Plasticity via Pretraining with Active Forgetting
Pretrained language models (PLMs) are today the primary model for natural
language processing. Despite their impressive downstream performance, it can be
difficult to apply PLMs to new languages, a barrier to making their
capabilities universally accessible. While prior work has shown it possible to
address this issue by learning a new embedding layer for the new language,
doing so is both data and compute inefficient. We propose to use an active
forgetting mechanism during pretraining, as a simple way of creating PLMs that
can quickly adapt to new languages. Concretely, by resetting the embedding
layer every K updates during pretraining, we encourage the PLM to improve its
ability of learning new embeddings within a limited number of updates, similar
to a meta-learning effect. Experiments with RoBERTa show that models pretrained
with our forgetting mechanism not only demonstrate faster convergence during
language adaptation but also outperform standard ones in a low-data regime,
particularly for languages that are distant from English.
Active forgetting mechanism during pretraining improves pretrained language models' adaptation to new, especially distant languages with limited data.
pretrained language models PLMs natural language processing embedding layer meta-learning language adaptation low-data regime