Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion
  Tokens
Are n-gram language models still relevant in this era of neural large
language models (LLMs)? Our answer is yes, and we show their values in both
text analysis and improving neural LLMs. Yet this necessitates modernizing
n-gram models in two aspects. First, we train them at the same data scale as
neural LLMs -- 1.4 trillion tokens. This is the largest n-gram model ever
built. Second, existing n-gram models use small n which hinders their
performance; we instead allow n to be arbitrarily large, by introducing a new
infty-gram LM with backoff. Instead of pre-computing n-gram count tables
(which would be very expensive), we develop an engine named infini-gram --
powered by suffix arrays -- that can compute infty-gram (as well as n-gram
with arbitrary n) probabilities with millisecond-level latency. The
infty-gram framework and infini-gram engine enable us to conduct many novel
and interesting analyses of human-written and machine-generated text: we find
that the infty-gram LM has fairly high accuracy for next-token prediction
(47%), and can complement neural LLMs to greatly reduce their language modeling
perplexities. When analyzing machine-generated text, we also observe
irregularities in the machine--infty-gram agreement level with respect to
the suffix length, which indicates deficiencies in neural LLM pretraining and
the positional embeddings of Transformers. We open-source our infini-gram
engine in the hopes of enabling more study on how to best use verbatim
information retrieved from large text corpora.
65baf35d87b68ba54f65c487
Jiacheng Liu
admin_assigned
2024-02-01T09:30:14.988Z
635f46d1928a42bc95cfcf7c
https://cdn-avatars.huggingface.co/v1/production/uploads/635f46d1928a42bc95cfcf7c/5KF8aLiDCJdl7B1SdJ-7V.png
Jiacheng Liu
user
liujch1998
65baf35d87b68ba54f65c488
Sewon Min
admin_assigned
2024-02-01T09:33:54.936Z
63a76d0de27a6dbd485fe863
https://cdn-avatars.huggingface.co/v1/production/uploads/63a76d0de27a6dbd485fe863/qJJwHOuvyQGq1o0KscOF_.jpeg
Sewon Min
user
sewon
65baf35d87b68ba54f65c489
Luke Zettlemoyer
65baf35d87b68ba54f65c48a
Yejin Choi
admin_assigned
2024-02-01T09:34:24.458Z
64d42729f63b01b7f676b176
/avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
Yejin Choi
user
yejinchoinka
65baf35d87b68ba54f65c48b
Hannaneh Hajishirzi
Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion
  Tokens
Are n-gram language models still relevant in this era of neural large
language models (LLMs)? Our answer is yes, and we show their values in both
text analysis and improving neural LLMs. Yet this necessitates modernizing
n-gram models in two aspects. First, we train them at the same data scale as
neural LLMs -- 1.4 trillion tokens. This is the largest n-gram model ever
built. Second, existing n-gram models use small n which hinders their
performance; we instead allow n to be arbitrarily large, by introducing a new
infty-gram LM with backoff. Instead of pre-computing n-gram count tables
(which would be very expensive), we develop an engine named infini-gram --
powered by suffix arrays -- that can compute infty-gram (as well as n-gram
with arbitrary n) probabilities with millisecond-level latency. The
infty-gram framework and infini-gram engine enable us to conduct many novel
and interesting analyses of human-written and machine-generated text: we find
that the infty-gram LM has fairly high accuracy for next-token prediction
(47%), and can complement neural LLMs to greatly reduce their language modeling
perplexities. When analyzing machine-generated text, we also observe
irregularities in the machine--infty-gram agreement level with respect to
the suffix length, which indicates deficiencies in neural LLM pretraining and
the positional embeddings of Transformers. We open-source our infini-gram
engine in the hopes of enabling more study on how to best use verbatim
information retrieved from large text corpora.
Modernized n-gram models, specifically $\infty$-gram models, enhance text analysis and neural LLMs by providing high accuracy in next-token prediction and reducing language modeling perplexities.
$\infty$-gram n-gram language modeling perplexity suffix arrays infini-gram engine Transformers