LM-Infinite: Simple On-the-Fly Length Generalization for Large Language
  Models
In recent years, there have been remarkable advancements in the performance
of Transformer-based Large Language Models (LLMs) across various domains. As
these LLMs are deployed for increasingly complex tasks, they often face the
needs to conduct longer reasoning processes or understanding larger contexts.
In these situations, the length generalization failure of LLMs on long
sequences become more prominent. Most pre-training schemes truncate training
sequences to a fixed length (such as 2048 for LLaMa). LLMs often struggle to
generate fluent texts, let alone carry out downstream tasks, after longer
contexts, even with relative positional encoding which is designed to cope with
this problem. Common solutions such as finetuning on longer corpora often
involves daunting hardware and time costs and requires careful training process
design. To more efficiently leverage the generation capacity of existing LLMs,
we theoretically and empirically investigate the main out-of-distribution (OOD)
factors contributing to this problem. Inspired by this diagnosis, we propose a
simple yet effective solution for on-the-fly length generalization,
LM-Infinite, which involves only a Lambda-shaped attention mask and a
distance limit while requiring no parameter updates or learning. We find it
applicable to a variety of LLMs using relative-position encoding methods.
LM-Infinite is computational efficient with O(n) time and space, and
demonstrates consistent fluency and generation quality to as long as 32k tokens
on ArXiv and OpenWebText2 datasets, with 2.72x decoding speedup. On downstream
task such as passkey retrieval, it continues to work on inputs much longer than
training lengths where vanilla models fail immediately.
64f0077f50dae59768e5bd9a
Chi Han
claimed_verified
2023-08-31T14:45:31.933Z
645b11ff5987734177a7cdcf
/avatars/3b04d2ca166eb28f227fc8d800e55c1c.svg
Chi Han
user
Glaciohound
64f0077f50dae59768e5bd9b
Qifan Wang
admin_assigned
2023-08-31T12:07:44.613Z
64b65c84c991daa61404abd7
/avatars/3592aa8774ea4e2836099e0701b4b78d.svg
WangQifan
user
Bayees
64f0077f50dae59768e5bd9c
Wenhan Xiong
64f0077f50dae59768e5bd9d
Yu Chen
claimed_verified
2023-09-11T06:54:47.451Z
647e90b31a1fcad2fdc52eb6
/avatars/d389261a14c3262fd95e241e66c44530.svg
Hugo Chen
user
hugochen
64f0077f50dae59768e5bd9e
Heng Ji
64f0077f50dae59768e5bd9f
Sinong Wang
LM-Infinite: Simple On-the-Fly Length Generalization for Large Language
  Models
In recent years, there have been remarkable advancements in the performance
of Transformer-based Large Language Models (LLMs) across various domains. As
these LLMs are deployed for increasingly complex tasks, they often face the
needs to conduct longer reasoning processes or understanding larger contexts.
In these situations, the length generalization failure of LLMs on long
sequences become more prominent. Most pre-training schemes truncate training
sequences to a fixed length (such as 2048 for LLaMa). LLMs often struggle to
generate fluent texts, let alone carry out downstream tasks, after longer
contexts, even with relative positional encoding which is designed to cope with
this problem. Common solutions such as finetuning on longer corpora often
involves daunting hardware and time costs and requires careful training process
design. To more efficiently leverage the generation capacity of existing LLMs,
we theoretically and empirically investigate the main out-of-distribution (OOD)
factors contributing to this problem. Inspired by this diagnosis, we propose a
simple yet effective solution for on-the-fly length generalization,
LM-Infinite, which involves only a Lambda-shaped attention mask and a
distance limit while requiring no parameter updates or learning. We find it
applicable to a variety of LLMs using relative-position encoding methods.
LM-Infinite is computational efficient with O(n) time and space, and
demonstrates consistent fluency and generation quality to as long as 32k tokens
on ArXiv and OpenWebText2 datasets, with 2.72x decoding speedup. On downstream
task such as passkey retrieval, it continues to work on inputs much longer than
training lengths where vanilla models fail immediately.
LM-Infinite addresses length generalization in Transformer-based LLMs using a simple $\Lambda$-shaped attention mask, achieving fluency and quality up to 32k tokens without retraining.
Transformer-based Large Language Models LLMs length generalization truncated training sequences relative positional encoding out-of-distribution OOD $\Lambda$-shaped attention mask computational efficiency ArXiv OpenWebText2 passkey retrieval