From Hours to Minutes: Lossless Acceleration of Ultra Long Sequence
  Generation up to 100K Tokens
Generating ultra-long sequences with large language models (LLMs) has become
increasingly crucial but remains a highly time-intensive task, particularly for
sequences up to 100K tokens. While traditional speculative decoding methods
exist, simply extending their generation limits fails to accelerate the process
and can be detrimental. Through an in-depth analysis, we identify three major
challenges hindering efficient generation: frequent model reloading, dynamic
key-value (KV) management and repetitive generation. To address these issues,
we introduce TOKENSWIFT, a novel framework designed to substantially accelerate
the generation process of ultra-long sequences while maintaining the target
model's inherent quality. Experimental results demonstrate that TOKENSWIFT
achieves over 3 times speedup across models of varying scales (1.5B, 7B, 8B,
14B) and architectures (MHA, GQA). This acceleration translates to hours of
time savings for ultra-long sequence generation, establishing TOKENSWIFT as a
scalable and effective solution at unprecedented lengths. Code can be found at
https://github.com/bigai-nlco/TokenSwift.
67c6cbd6e52534aa6ada2e26
Tong Wu
admin_assigned
2025-03-04T10:58:45.670Z
668f7fee5156d55f72af4f21
/avatars/02edf8d7d5f288d80dc665b18dda4d0a.svg
TongWu
user
TongWu
67c6cbd6e52534aa6ada2e27
Junzhe Shen
admin_assigned
2025-03-04T10:58:27.834Z
6530c9d7d107f378e105d667
/avatars/889dfcb6514c90351802bebb4a34a78f.svg
Junzhe Shen
user
JunzheS
67c6cbd6e52534aa6ada2e28
Zixia Jia
admin_assigned
2025-03-04T10:58:34.128Z
64b7ae6cf53ae848e72b997d
/avatars/b55dd3d6fcb3ccac2e3880d01a9bdc63.svg
Zixia Jia
user
vickyandkekey
67c6cbd6e52534aa6ada2e29
Yuxuan Wang
claimed_verified
2025-03-05T10:12:27.432Z
60b9e6837946aff342f734ae
/avatars/a711a6aa35757dfd7b78b26098a964fc.svg
Yuxuan Wang
user
ColorfulAI
67c6cbd6e52534aa6ada2e2a
Zilong Zheng
extracted_pending
2025-03-04T09:45:59.571Z
63a95a6a7930fa8c7dd63d4e
/avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg
Zilong Zheng
user
zlzheng
From Hours to Minutes: Lossless Acceleration of Ultra Long Sequence
  Generation up to 100K Tokens
Generating ultra-long sequences with large language models (LLMs) has become
increasingly crucial but remains a highly time-intensive task, particularly for
sequences up to 100K tokens. While traditional speculative decoding methods
exist, simply extending their generation limits fails to accelerate the process
and can be detrimental. Through an in-depth analysis, we identify three major
challenges hindering efficient generation: frequent model reloading, dynamic
key-value (KV) management and repetitive generation. To address these issues,
we introduce TOKENSWIFT, a novel framework designed to substantially accelerate
the generation process of ultra-long sequences while maintaining the target
model's inherent quality. Experimental results demonstrate that TOKENSWIFT
achieves over 3 times speedup across models of varying scales (1.5B, 7B, 8B,
14B) and architectures (MHA, GQA). This acceleration translates to hours of
time savings for ultra-long sequence generation, establishing TOKENSWIFT as a
scalable and effective solution at unprecedented lengths. Code can be found at
https://github.com/bigai-nlco/TokenSwift.
TOKENSWIFT addresses challenges in ultra-long sequence generation by introducing a framework that achieves significant speedup without compromising model quality.
large language models LLMS speculative decoding model reloading key-value management TOKENSWIFT ultra-long sequences time-intensive task multi-head attention GQA