Language Modeling Is Compression
It has long been established that predictive models can be transformed into
lossless compressors and vice versa. Incidentally, in recent years, the machine
learning community has focused on training increasingly large and powerful
self-supervised (language) models. Since these large language models exhibit
impressive predictive capabilities, they are well-positioned to be strong
compressors. In this work, we advocate for viewing the prediction problem
through the lens of compression and evaluate the compression capabilities of
large (foundation) models. We show that large language models are powerful
general-purpose predictors and that the compression viewpoint provides novel
insights into scaling laws, tokenization, and in-context learning. For example,
Chinchilla 70B, while trained primarily on text, compresses ImageNet patches to
43.4% and LibriSpeech samples to 16.4% of their raw size, beating
domain-specific compressors like PNG (58.5%) or FLAC (30.3%), respectively.
Finally, we show that the prediction-compression equivalence allows us to use
any compressor (like gzip) to build a conditional generative model.
650a50b9d48ed98e63cebb86
Grégoire Delétang
650a50b9d48ed98e63cebb87
Anian Ruoss
claimed_verified
2024-02-09T08:06:43.296Z
65c51758522ee3a4edd917d0
/avatars/2ea3b64b5eeb34e2bfe821e4210f1b3e.svg
Anian Ruoss
user
anianruoss
650a50b9d48ed98e63cebb88
Paul-Ambroise Duquenne
admin_assigned
2023-09-20T10:28:38.106Z
6331a09fe092098b57bc1d83
/avatars/cdbc4be17c160b5e1f957b6c1ae29e22.svg
Paul-Ambroise Duquenne
user
padqn
650a50b9d48ed98e63cebb89
Elliot Catt
650a50b9d48ed98e63cebb8a
Tim Genewein
650a50b9d48ed98e63cebb8b
Christopher Mattern
650a50b9d48ed98e63cebb8c
Jordi Grau-Moya
650a50b9d48ed98e63cebb8d
Li Kevin Wenliang
650a50b9d48ed98e63cebb8e
Matthew Aitchison
650a50b9d48ed98e63cebb8f
Laurent Orseau
admin_assigned
2023-09-20T11:28:01.174Z
64ae74ad71b7e849b0aa3404
/avatars/50585fdd3fc2a916814c500ca6bb3c00.svg
Laurent Orseau
user
lorseau
650a50b9d48ed98e63cebb90
Marcus Hutter
650a50b9d48ed98e63cebb91
Joel Veness
Language Modeling Is Compression
It has long been established that predictive models can be transformed into
lossless compressors and vice versa. Incidentally, in recent years, the machine
learning community has focused on training increasingly large and powerful
self-supervised (language) models. Since these large language models exhibit
impressive predictive capabilities, they are well-positioned to be strong
compressors. In this work, we advocate for viewing the prediction problem
through the lens of compression and evaluate the compression capabilities of
large (foundation) models. We show that large language models are powerful
general-purpose predictors and that the compression viewpoint provides novel
insights into scaling laws, tokenization, and in-context learning. For example,
Chinchilla 70B, while trained primarily on text, compresses ImageNet patches to
43.4% and LibriSpeech samples to 16.4% of their raw size, beating
domain-specific compressors like PNG (58.5%) or FLAC (30.3%), respectively.
Finally, we show that the prediction-compression equivalence allows us to use
any compressor (like gzip) to build a conditional generative model.
Large language models demonstrate strong compression capabilities, outperforming domain-specific compressors and offering new insights into scaling laws, tokenization, and in-context learning.
self-supervised models large language models compression Chinchilla ImageNet LibriSpeech PNG FLAC gzip conditional generative model