Shifting AI Efficiency From Model-Centric to Data-Centric Compression
The rapid advancement of large language models (LLMs) and multi-modal LLMs
(MLLMs) has historically relied on model-centric scaling through increasing
parameter counts from millions to hundreds of billions to drive performance
gains. However, as we approach hardware limits on model size, the dominant
computational bottleneck has fundamentally shifted to the quadratic cost of
self-attention over long token sequences, now driven by ultra-long text
contexts, high-resolution images, and extended videos. In this position paper,
we argue that the focus of research for efficient AI is shifting from
model-centric compression to data-centric compression. We position token
compression as the new frontier, which improves AI efficiency via reducing the
number of tokens during model training or inference. Through comprehensive
analysis, we first examine recent developments in long-context AI across
various domains and establish a unified mathematical framework for existing
model efficiency strategies, demonstrating why token compression represents a
crucial paradigm shift in addressing long-context overhead. Subsequently, we
systematically review the research landscape of token compression, analyzing
its fundamental benefits and identifying its compelling advantages across
diverse scenarios. Furthermore, we provide an in-depth analysis of current
challenges in token compression research and outline promising future
directions. Ultimately, our work aims to offer a fresh perspective on AI
efficiency, synthesize existing research, and catalyze innovative developments
to address the challenges that increasing context lengths pose to the AI
community's advancement.
68353258d005e45149d2d384
Xuyang Liu
claimed_verified
2025-05-27T08:23:05.932Z
66a0caa1a7a6ed88ad1c0ddf
https://cdn-avatars.huggingface.co/v1/production/uploads/66a0caa1a7a6ed88ad1c0ddf/WoOP24-ruuHy4ryNhRp0D.jpeg
Xuyang Liu
user
xuyang-liu16
68353258d005e45149d2d385
Zichen Wen
claimed_verified
2025-05-27T07:50:45.710Z
653b8c3e97a4d71d950e2f20
/avatars/b68880022e14556d0be58c69615db3be.svg
Zichen Wen
user
zichenwen
68353258d005e45149d2d386
Shaobo Wang
claimed_verified
2025-05-27T07:50:41.853Z
66968099c952e09a4cb29f78
/avatars/bd3a361fe5315e26e9ae328071704eed.svg
Wang
user
Steven-Shaobo
68353258d005e45149d2d387
Junjie Chen
claimed_verified
2025-05-27T07:50:48.148Z
652f8642338c761caf474169
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/mq5jjqqNaFxVboWGDEocJ.jpeg
Junjie Chen
user
coderchen01
68353258d005e45149d2d388
Zhishan Tao
claimed_verified
2025-05-27T07:50:43.758Z
679f280ffb07d74f084520b6
/avatars/b378000f68c7faf8d4fee8074dd2db5b.svg
Zhishan Tao
user
Pppeach33
68353258d005e45149d2d389
Yubo Wang
68353258d005e45149d2d38a
Xiangqi Jin
admin_assigned
2025-05-27T08:21:21.961Z
64abcbfde144ba0eb9bb8419
/avatars/6ccea0e755bad384aaabd5c455bd962e.svg
Xiangqi Jin
user
Lueci4er
68353258d005e45149d2d38b
Chang Zou
claimed_verified
2025-07-21T08:31:06.699Z
6623417a9c9b00aaff64de33
/avatars/58c108ebf486d90866d8626bbb69312d.svg
Chang Zou
user
Shenyizou
68353258d005e45149d2d38c
Yiyu Wang
68353258d005e45149d2d38d
Chenfei Liao
claimed_verified
2025-05-28T09:00:32.765Z
6806464ed918f6d2fee2bc8b
https://cdn-avatars.huggingface.co/v1/production/uploads/6806464ed918f6d2fee2bc8b/rgpG2oO0m6PT0KltCF_Wf.jpeg
Chenfei Liao
user
Chenfei-Liao
68353258d005e45149d2d38e
Xu Zheng
claimed_verified
2025-06-12T12:43:38.659Z
65961a7daa677adc97d8e33f
https://cdn-avatars.huggingface.co/v1/production/uploads/65961a7daa677adc97d8e33f/TA0Vz7CDmYZEdrhnsj_1b.jpeg
Xu Zheng
user
xz287
68353258d005e45149d2d38f
Honggang Chen
68353258d005e45149d2d390
Weijia Li
68353258d005e45149d2d391
Xuming Hu
68353258d005e45149d2d392
Conghui He
admin_assigned
2025-05-27T08:22:15.329Z
63f9fca8d4349b157a109eec
/avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
Conghui He
user
conghui
68353258d005e45149d2d393
Linfeng Zhang
admin_assigned
2025-05-27T08:22:07.787Z
642ec9831d1737803dc1c30a
/avatars/c9ded838bad09004c15a27200e66a108.svg
linfeng zhang
user
linfengZ
Shifting AI Efficiency From Model-Centric to Data-Centric Compression
The rapid advancement of large language models (LLMs) and multi-modal LLMs
(MLLMs) has historically relied on model-centric scaling through increasing
parameter counts from millions to hundreds of billions to drive performance
gains. However, as we approach hardware limits on model size, the dominant
computational bottleneck has fundamentally shifted to the quadratic cost of
self-attention over long token sequences, now driven by ultra-long text
contexts, high-resolution images, and extended videos. In this position paper,
we argue that the focus of research for efficient AI is shifting from
model-centric compression to data-centric compression. We position token
compression as the new frontier, which improves AI efficiency via reducing the
number of tokens during model training or inference. Through comprehensive
analysis, we first examine recent developments in long-context AI across
various domains and establish a unified mathematical framework for existing
model efficiency strategies, demonstrating why token compression represents a
crucial paradigm shift in addressing long-context overhead. Subsequently, we
systematically review the research landscape of token compression, analyzing
its fundamental benefits and identifying its compelling advantages across
diverse scenarios. Furthermore, we provide an in-depth analysis of current
challenges in token compression research and outline promising future
directions. Ultimately, our work aims to offer a fresh perspective on AI
efficiency, synthesize existing research, and catalyze innovative developments
to address the challenges that increasing context lengths pose to the AI
community's advancement.
The focus in AI research is shifting from model-centric to data-centric compression, with token compression identified as key to improving efficiency in handling long-context scenarios.
large language models multi-modal LLMs self-attention token compression long-context AI mathematical framework model efficiency long-context overhead current challenges future directions