Block Transformer: Global-to-Local Language Modeling for Fast Inference
This paper presents the Block Transformer architecture which adopts
hierarchical global-to-local modeling to autoregressive transformers to
mitigate the inference bottlenecks of self-attention. To apply self-attention,
the key-value (KV) cache of all previous sequences must be retrieved from
memory at every decoding step. Thereby, this KV cache IO becomes a significant
bottleneck in batch inference. We notice that these costs stem from applying
self-attention on the global context, therefore we isolate the expensive
bottlenecks of global modeling to lower layers and apply fast local modeling in
upper layers. To mitigate the remaining costs in the lower layers, we aggregate
input tokens into fixed size blocks and then apply self-attention at this
coarse level. Context information is aggregated into a single embedding to
enable upper layers to decode the next block of tokens, without global
attention. Free of global attention bottlenecks, the upper layers can fully
utilize the compute hardware to maximize inference throughput. By leveraging
global and local modules, the Block Transformer architecture demonstrates
10-20x gains in inference throughput compared to vanilla transformers with
equivalent perplexity. Our work introduces a new approach to optimize language
model inference through novel application of global-to-local modeling. Code is
available at https://github.com/itsnamgyu/block-transformer.
666120834e5af73c6a4ea91d
Namgyu Ho
admin_assigned
2024-06-06T08:13:42.601Z
62d0f7faad741b94f5d13744
https://cdn-avatars.huggingface.co/v1/production/uploads/1677826870505-62d0f7faad741b94f5d13744.jpeg
Namgyu Ho
user
itsnamgyu
666120834e5af73c6a4ea91e
Sangmin Bae
admin_assigned
2024-06-06T08:13:59.532Z
6602ca1e10a1441af41637be
/avatars/5880e699def320beb352cbed77495b2f.svg
Sangmin Bae
user
raymin0223
666120834e5af73c6a4ea91f
Taehyeon Kim
admin_assigned
2024-06-06T08:14:45.383Z
639f32952e13e54dcbb9bf8d
/avatars/a797b72363e06cd2222165493bf7e39c.svg
Taehyeon Kim
user
Kthyeon
666120834e5af73c6a4ea920
Hyunjik Jo
666120834e5af73c6a4ea921
Yireun Kim
admin_assigned
2024-06-06T08:20:39.322Z
660260cf1737e5cd4a826550
https://cdn-avatars.huggingface.co/v1/production/uploads/660260cf1737e5cd4a826550/AlSfoM2WtqPjLtYR6x7Wf.jpeg
Yireun Kim
user
yireun
666120834e5af73c6a4ea922
Tal Schuster
admin_assigned
2024-06-06T08:20:45.770Z
60440983efd035140b99c814
/avatars/4afb3cc5b940b24bd2794d07f694b9ef.svg
Tal Schuster
user
tals
666120834e5af73c6a4ea923
Adam Fisch
666120834e5af73c6a4ea924
James Thorne
claimed_verified
2024-06-07T09:20:34.411Z
63b54db5889aa6707f08f2d1
https://cdn-avatars.huggingface.co/v1/production/uploads/63b54db5889aa6707f08f2d1/zUnyx-8Zabxr54poZtUo4.jpeg
James Thorne
user
j6mes
666120834e5af73c6a4ea925
Se-Young Yun
Block Transformer: Global-to-Local Language Modeling for Fast Inference
This paper presents the Block Transformer architecture which adopts
hierarchical global-to-local modeling to autoregressive transformers to
mitigate the inference bottlenecks of self-attention. To apply self-attention,
the key-value (KV) cache of all previous sequences must be retrieved from
memory at every decoding step. Thereby, this KV cache IO becomes a significant
bottleneck in batch inference. We notice that these costs stem from applying
self-attention on the global context, therefore we isolate the expensive
bottlenecks of global modeling to lower layers and apply fast local modeling in
upper layers. To mitigate the remaining costs in the lower layers, we aggregate
input tokens into fixed size blocks and then apply self-attention at this
coarse level. Context information is aggregated into a single embedding to
enable upper layers to decode the next block of tokens, without global
attention. Free of global attention bottlenecks, the upper layers can fully
utilize the compute hardware to maximize inference throughput. By leveraging
global and local modules, the Block Transformer architecture demonstrates
10-20x gains in inference throughput compared to vanilla transformers with
equivalent perplexity. Our work introduces a new approach to optimize language
model inference through novel application of global-to-local modeling. Code is
available at https://github.com/itsnamgyu/block-transformer.
The Block Transformer architecture enhances inference throughput by applying global-to-local modeling to autoregressive transformers, reducing inference bottlenecks through hierarchical processing and block-level self-attention.
Block Transformer hierarchical global-to-local modeling autoregressive transformers self-attention key-value cache token aggregation embedding inference throughput global attention