Transformers Can Achieve Length Generalization But Not Robustly
Length generalization, defined as the ability to extrapolate from shorter
training sequences to longer test ones, is a significant challenge for language
models. This issue persists even with large-scale Transformers handling
relatively straightforward tasks. In this paper, we test the Transformer's
ability of length generalization using the task of addition of two integers. We
show that the success of length generalization is intricately linked to the
data format and the type of position encoding. Using the right combination of
data format and position encodings, we show for the first time that standard
Transformers can extrapolate to a sequence length that is 2.5x the input
length. Nevertheless, unlike in-distribution generalization, length
generalization remains fragile, significantly influenced by factors like random
weight initialization and training data order, leading to large variances
across different random seeds.
65cd97177b7a844001f5d803
Yongchao Zhou
admin_assigned
2024-02-15T09:54:19.590Z
60df063fbb3e1aaa18b005bc
/avatars/f7210ffd942c124266845bc4d4d89148.svg
Yongchao Zhou
user
zycluke
65cd97177b7a844001f5d804
Uri Alon
admin_assigned
2024-02-15T09:54:36.778Z
62cd8153248f9e6bc20ab250
https://cdn-avatars.huggingface.co/v1/production/uploads/1657635637504-62cd8153248f9e6bc20ab250.jpeg
Uri Alon
user
urialon
65cd97177b7a844001f5d805
Xinyun Chen
admin_assigned
2024-02-15T09:55:04.137Z
64d0268001931c60161e026a
/avatars/0ea48c47b0270b449fd6b97b495e64a6.svg
Xinyun Chen
user
xinyunchen
65cd97177b7a844001f5d806
Xuezhi Wang
65cd97177b7a844001f5d807
Rishabh Agarwal
admin_assigned
2024-02-15T09:55:37.591Z
65318c1022350ffeaf7726ef
/avatars/9359e63d38cb69d08da2de07ae933885.svg
Rishabh Agarwal
user
rishabhagarwal
65cd97177b7a844001f5d808
Denny Zhou
claimed_verified
2024-02-20T18:26:33.392Z
65d4d62d493aeaf0bb295f1b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vXMxaoWwFziul8TqIhQw8.jpeg
Denny Zhou
user
dennyzhou
Transformers Can Achieve Length Generalization But Not Robustly
Length generalization, defined as the ability to extrapolate from shorter
training sequences to longer test ones, is a significant challenge for language
models. This issue persists even with large-scale Transformers handling
relatively straightforward tasks. In this paper, we test the Transformer's
ability of length generalization using the task of addition of two integers. We
show that the success of length generalization is intricately linked to the
data format and the type of position encoding. Using the right combination of
data format and position encodings, we show for the first time that standard
Transformers can extrapolate to a sequence length that is 2.5x the input
length. Nevertheless, unlike in-distribution generalization, length
generalization remains fragile, significantly influenced by factors like random
weight initialization and training data order, leading to large variances
across different random seeds.
Transformers can extrapolate to longer test sequences with the right data format and position encodings, but length generalization is sensitive to initialization and data order.
Transformers length generalization addition task position encodings in-distribution generalization