A Silver Bullet or a Compromise for Full Attention? A Comprehensive
  Study of Gist Token-based Context Compression
In this work, we provide a thorough investigation of gist-based context
compression methods to improve long-context processing in large language
models. We focus on two key questions: (1) How well can these methods replace
full attention models? and (2) What potential failure patterns arise due to
compression? Through extensive experiments, we show that while gist-based
compression can achieve near-lossless performance on tasks like
retrieval-augmented generation and long-document QA, it faces challenges in
tasks like synthetic recall. Furthermore, we identify three key failure
patterns: lost by the boundary, lost if surprise, and lost along the way. To
mitigate these issues, we propose two effective strategies: fine-grained
autoencoding, which enhances the reconstruction of original token information,
and segment-wise token importance estimation, which adjusts optimization based
on token dependencies. Our work provides valuable insights into the
understanding of gist token-based context compression and offers practical
strategies for improving compression capabilities.
676a2a3ebce62ec5a02c4a66
Chenlong Deng
admin_assigned
2024-12-26T18:49:33.308Z
654c99d6e82a71cb487c2ecd
/avatars/480c883b587ed2e41e1e9661c844a938.svg
ChenlongDeng
user
ChenlongDeng
676a2a3ebce62ec5a02c4a67
Zhisong Zhang
676a2a3ebce62ec5a02c4a68
Kelong Mao
claimed_verified
2024-12-30T19:35:30.975Z
649a65605c74a2125c22bbc1
/avatars/e7435d3aeeb59acc6f6f43b48d6982a0.svg
Mao
user
kyriemao
676a2a3ebce62ec5a02c4a69
Shuaiyi Li
claimed_verified
2024-12-30T19:35:28.792Z
64d708e45e5f05485ce5fbd8
/avatars/3c5389a5ffecbc05583004c627b21b6c.svg
Li Shuaiyi
user
Syon-Li
676a2a3ebce62ec5a02c4a6a
Xinting Huang
676a2a3ebce62ec5a02c4a6b
Dong Yu
676a2a3ebce62ec5a02c4a6c
Zhicheng Dou
A Silver Bullet or a Compromise for Full Attention? A Comprehensive
  Study of Gist Token-based Context Compression
In this work, we provide a thorough investigation of gist-based context
compression methods to improve long-context processing in large language
models. We focus on two key questions: (1) How well can these methods replace
full attention models? and (2) What potential failure patterns arise due to
compression? Through extensive experiments, we show that while gist-based
compression can achieve near-lossless performance on tasks like
retrieval-augmented generation and long-document QA, it faces challenges in
tasks like synthetic recall. Furthermore, we identify three key failure
patterns: lost by the boundary, lost if surprise, and lost along the way. To
mitigate these issues, we propose two effective strategies: fine-grained
autoencoding, which enhances the reconstruction of original token information,
and segment-wise token importance estimation, which adjusts optimization based
on token dependencies. Our work provides valuable insights into the
understanding of gist token-based context compression and offers practical
strategies for improving compression capabilities.
Gist-based context compression in large language models offers near-lossless performance on some tasks but fails on synthetic recall, with identified failure patterns and proposed strategies for improvement.
gist-based context compression long-context processing large language models full attention models retrieval-augmented generation long-document QA synthetic recall fine-grained autoencoding segment-wise token importance estimation