Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the
  Limits of Embedding Space Capacity
A range of recent works addresses the problem of compression of sequence of
tokens into a shorter sequence of real-valued vectors to be used as inputs
instead of token embeddings or key-value cache. These approaches allow to
reduce the amount of compute in existing language models. Despite relying on
powerful models as encoders, the maximum attainable lossless compression ratio
is typically not higher than x10. This fact is highly intriguing because, in
theory, the maximum information capacity of large real-valued vectors is far
beyond the presented rates even for 16-bit precision and a modest vector size.
In this work, we explore the limits of compression by replacing the encoder
with a per-sample optimization procedure. We show that vectors with compression
ratios up to x1500 exist, which highlights two orders of magnitude gap between
existing and practically attainable solutions. Furthermore, we empirically show
that the compression limits are determined not by the length of the input but
by the amount of uncertainty to be reduced, namely, the cross-entropy loss on
this sequence without any conditioning. The obtained limits highlight the
substantial gap between the theoretical capacity of input embeddings and their
practical utilization, suggesting significant room for optimization in model
design.
67b5a7896f72266cb765e744
Yuri Kuratov
extracted_confirmed
2025-06-06T17:43:24.656Z
618b9540682ec1c38327e586
https://cdn-avatars.huggingface.co/v1/production/uploads/618b9540682ec1c38327e586/v_ZBkfh8O9Zh6C2YQpuBX.jpeg
Yury Kuratov
user
yurakuratov
67b5a7896f72266cb765e745
Mikhail Arkhipov
67b5a7896f72266cb765e746
Aydar Bulatov
claimed_verified
2025-02-21T14:43:06.400Z
64c8b321cb2f1bf0e7c0f54b
https://cdn-avatars.huggingface.co/v1/production/uploads/64c8b321cb2f1bf0e7c0f54b/JflXxMVnG9I0IB5YNyhXF.jpeg
Aydar Bulatov
user
booydar
67b5a7896f72266cb765e747
Mikhail Burtsev
claimed_verified
2025-02-19T09:56:59.080Z
639c6e978a34ed9a404c6a7b
/avatars/c98ca8c9f9ed8509c2f1bb6aa994fd57.svg
MIKHAIL BURTSEV
user
mbur
Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the
  Limits of Embedding Space Capacity
A range of recent works addresses the problem of compression of sequence of
tokens into a shorter sequence of real-valued vectors to be used as inputs
instead of token embeddings or key-value cache. These approaches allow to
reduce the amount of compute in existing language models. Despite relying on
powerful models as encoders, the maximum attainable lossless compression ratio
is typically not higher than x10. This fact is highly intriguing because, in
theory, the maximum information capacity of large real-valued vectors is far
beyond the presented rates even for 16-bit precision and a modest vector size.
In this work, we explore the limits of compression by replacing the encoder
with a per-sample optimization procedure. We show that vectors with compression
ratios up to x1500 exist, which highlights two orders of magnitude gap between
existing and practically attainable solutions. Furthermore, we empirically show
that the compression limits are determined not by the length of the input but
by the amount of uncertainty to be reduced, namely, the cross-entropy loss on
this sequence without any conditioning. The obtained limits highlight the
substantial gap between the theoretical capacity of input embeddings and their
practical utilization, suggesting significant room for optimization in model
design.
Using per-sample optimization, compression ratios of up to x1500 are achieved for sequence-to-vector compression in language models, highlighting a large gap between theoretical and practical limits.
sequence compression token embeddings per-sample optimization compression ratio cross-entropy loss model design