Compression Represents Intelligence Linearly
There is a belief that learning to compress well will lead to intelligence.
Recently, language modeling has been shown to be equivalent to compression,
which offers a compelling rationale for the success of large language models
(LLMs): the development of more advanced language models is essentially
enhancing compression which facilitates intelligence. Despite such appealing
discussions, little empirical evidence is present for the interplay between
compression and intelligence. In this work, we examine their relationship in
the context of LLMs, treating LLMs as data compressors. Given the abstract
concept of "intelligence", we adopt the average downstream benchmark scores as
a surrogate, specifically targeting intelligence related to knowledge and
commonsense, coding, and mathematical reasoning. Across 12 benchmarks, our
study brings together 30 public LLMs that originate from diverse organizations.
Remarkably, we find that LLMs' intelligence -- reflected by average benchmark
scores -- almost linearly correlates with their ability to compress external
text corpora. These results provide concrete evidence supporting the belief
that superior compression indicates greater intelligence. Furthermore, our
findings suggest that compression efficiency, as an unsupervised metric derived
from raw text corpora, serves as a reliable evaluation measure that is linearly
associated with the model capabilities. We open-source our compression datasets
as well as our data collection pipelines to facilitate future researchers to
assess compression properly.
661df263e5094183cd1008d8
Yuzhen Huang
claimed_verified
2024-04-16T06:27:09.152Z
6462def82a83863b97c0611e
/avatars/c03e9cc7d75b0266fcc56ecb6ee62148.svg
Yuzhen Huang
user
yuzhen17
661df263e5094183cd1008d9
Jinghan Zhang
claimed_verified
2024-04-18T19:35:11.700Z
642c5631ab0cc792e4379099
/avatars/c660650950d4804f48329970226e63d1.svg
Jinghan ZHANG
user
jinghan23
661df263e5094183cd1008da
Zifei Shan
661df263e5094183cd1008db
Junxian He
admin_assigned
2024-04-18T09:42:16.631Z
615f34ec3f6d24d67c1b5c78
/avatars/6dcff6477993d9e57c5cb92b6f95eb66.svg
Junxian He
user
jxhe
Compression Represents Intelligence Linearly
There is a belief that learning to compress well will lead to intelligence.
Recently, language modeling has been shown to be equivalent to compression,
which offers a compelling rationale for the success of large language models
(LLMs): the development of more advanced language models is essentially
enhancing compression which facilitates intelligence. Despite such appealing
discussions, little empirical evidence is present for the interplay between
compression and intelligence. In this work, we examine their relationship in
the context of LLMs, treating LLMs as data compressors. Given the abstract
concept of "intelligence", we adopt the average downstream benchmark scores as
a surrogate, specifically targeting intelligence related to knowledge and
commonsense, coding, and mathematical reasoning. Across 12 benchmarks, our
study brings together 30 public LLMs that originate from diverse organizations.
Remarkably, we find that LLMs' intelligence -- reflected by average benchmark
scores -- almost linearly correlates with their ability to compress external
text corpora. These results provide concrete evidence supporting the belief
that superior compression indicates greater intelligence. Furthermore, our
findings suggest that compression efficiency, as an unsupervised metric derived
from raw text corpora, serves as a reliable evaluation measure that is linearly
associated with the model capabilities. We open-source our compression datasets
as well as our data collection pipelines to facilitate future researchers to
assess compression properly.
Research shows that the ability of large language models to compress text is linearly associated with their performance on various benchmarks related to intelligence, suggesting compression efficiency as a reliable evaluation metric.
large language models compression benchmarks knowledge commonsense coding mathematical reasoning compression datasets