Zebra: Extending Context Window with Layerwise Grouped Local-Global
  Attention
This paper introduces a novel approach to enhance the capabilities of Large
Language Models (LLMs) in processing and understanding extensive text
sequences, a critical aspect in applications requiring deep comprehension and
synthesis of large volumes of information. Recognizing the inherent challenges
in extending the context window for LLMs, primarily built on Transformer
architecture, we propose a new model architecture, referred to as Zebra. This
architecture efficiently manages the quadratic time and memory complexity
issues associated with full attention in the Transformer by employing grouped
local-global attention layers. Our model, akin to a zebra's alternating
stripes, balances local and global attention layers, significantly reducing
computational requirements and memory consumption. Comprehensive experiments,
including pretraining from scratch, continuation of long context adaptation
training, and long instruction tuning, are conducted to evaluate the Zebra's
performance. The results show that Zebra achieves comparable or superior
performance on both short and long sequence benchmarks, while also enhancing
training and inference efficiency.
657bc91afcba5f698c23000d
Kaiqiang Song
admin_assigned
2023-12-15T10:59:29.731Z
63a261f13c003e409322c191
/avatars/dc4be6ce1983d0d32620275df124fe45.svg
Kaiqiang Song
user
kqsong
657bc91afcba5f698c23000e
Xiaoyang Wang
claimed_verified
2023-12-18T08:48:16.037Z
657cd228138b7e391444a65d
/avatars/c7c984ae483144fab627aa2c54d91d0f.svg
Xiaoyang Wang
user
xywang1
657bc91afcba5f698c23000f
Sangwoo Cho
claimed_verified
2024-02-01T08:44:35.869Z
64dbc6251d19239f50752356
/avatars/3d60998db3dd8f37a5158e1758c0f706.svg
Sangwoo Cho
user
swcho3
657bc91afcba5f698c230010
Xiaoman Pan
657bc91afcba5f698c230011
Dong Yu
Zebra: Extending Context Window with Layerwise Grouped Local-Global
  Attention
This paper introduces a novel approach to enhance the capabilities of Large
Language Models (LLMs) in processing and understanding extensive text
sequences, a critical aspect in applications requiring deep comprehension and
synthesis of large volumes of information. Recognizing the inherent challenges
in extending the context window for LLMs, primarily built on Transformer
architecture, we propose a new model architecture, referred to as Zebra. This
architecture efficiently manages the quadratic time and memory complexity
issues associated with full attention in the Transformer by employing grouped
local-global attention layers. Our model, akin to a zebra's alternating
stripes, balances local and global attention layers, significantly reducing
computational requirements and memory consumption. Comprehensive experiments,
including pretraining from scratch, continuation of long context adaptation
training, and long instruction tuning, are conducted to evaluate the Zebra's
performance. The results show that Zebra achieves comparable or superior
performance on both short and long sequence benchmarks, while also enhancing
training and inference efficiency.
A new model architecture called Zebra addresses the challenges of handling large text sequences in LLMs by using grouped local-global attention layers, improving efficiency and performance.
Transformer architecture grouped local-global attention layers computational requirements memory consumption pretraining long context adaptation training long instruction tuning