Liger: Linearizing Large Language Models to Gated Recurrent Structures
Transformers with linear recurrent modeling offer linear-time training and
constant-memory inference. Despite their demonstrated efficiency and
performance, pretraining such non-standard architectures from scratch remains
costly and risky. The linearization of large language models (LLMs) transforms
pretrained standard models into linear recurrent structures, enabling more
efficient deployment. However, current linearization methods typically
introduce additional feature map modules that require extensive fine-tuning and
overlook the gating mechanisms used in state-of-the-art linear recurrent
models. To address these issues, this paper presents Liger, short for
Linearizing LLMs to gated recurrent structures. Liger is a novel approach for
converting pretrained LLMs into gated linear recurrent models without adding
extra parameters. It repurposes the pretrained key matrix weights to construct
diverse gating mechanisms, facilitating the formation of various gated
recurrent structures while avoiding the need to train additional components
from scratch. Using lightweight fine-tuning with Low-Rank Adaptation (LoRA),
Liger restores the performance of the linearized gated recurrent models to
match that of the original LLMs. Additionally, we introduce Liger Attention, an
intra-layer hybrid attention mechanism, which significantly recovers 93\% of
the Transformer-based LLM at 0.02\% pre-training tokens during the
linearization process, achieving competitive results across multiple
benchmarks, as validated on models ranging from 1B to 8B parameters. Code is
available at https://github.com/OpenSparseLLMs/Linearization.
67c6b05f35198d0f397adc98
Disen Lan
claimed_verified
2025-03-04T08:34:46.117Z
66ea643899af9ac3463639b1
/avatars/252d470e761a57834dee3dbc60dfefed.svg
Disen Lan
user
landisen
67c6b05f35198d0f397adc99
Weigao Sun
extracted_confirmed
2025-03-04T08:10:52.130Z
6246bb33da617c00b48e4d92
/avatars/0304a9f6eb7f5dee4d933d03222f94e9.svg
Weigao Sun
user
weigao266
67c6b05f35198d0f397adc9a
Jiaxi Hu
admin_assigned
2025-03-04T10:04:18.982Z
665dc35752ff9daa9ba5a4ed
/avatars/df8b01879d97e599b610fa51414d3a18.svg
Hu Jiaxi
user
Jiaxihu2
67c6b05f35198d0f397adc9b
Jusen Du
admin_assigned
2025-03-04T10:04:26.432Z
65003e857804f04a163328d9
/avatars/fe32150aabfde8d283b38ccebcf6982e.svg
Jusen Du
user
JusenK
67c6b05f35198d0f397adc9c
Yu Cheng
claimed_verified
2025-06-02T07:52:49.648Z
67017abfe4d49b157ac534d9
/avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
Yu Cheng
user
ych133
Liger: Linearizing Large Language Models to Gated Recurrent Structures
Transformers with linear recurrent modeling offer linear-time training and
constant-memory inference. Despite their demonstrated efficiency and
performance, pretraining such non-standard architectures from scratch remains
costly and risky. The linearization of large language models (LLMs) transforms
pretrained standard models into linear recurrent structures, enabling more
efficient deployment. However, current linearization methods typically
introduce additional feature map modules that require extensive fine-tuning and
overlook the gating mechanisms used in state-of-the-art linear recurrent
models. To address these issues, this paper presents Liger, short for
Linearizing LLMs to gated recurrent structures. Liger is a novel approach for
converting pretrained LLMs into gated linear recurrent models without adding
extra parameters. It repurposes the pretrained key matrix weights to construct
diverse gating mechanisms, facilitating the formation of various gated
recurrent structures while avoiding the need to train additional components
from scratch. Using lightweight fine-tuning with Low-Rank Adaptation (LoRA),
Liger restores the performance of the linearized gated recurrent models to
match that of the original LLMs. Additionally, we introduce Liger Attention, an
intra-layer hybrid attention mechanism, which significantly recovers 93\% of
the Transformer-based LLM at 0.02\% pre-training tokens during the
linearization process, achieving competitive results across multiple
benchmarks, as validated on models ranging from 1B to 8B parameters. Code is
available at https://github.com/OpenSparseLLMs/Linearization.
Liger, a novel method, converts pretrained large language models into gated linear recurrent structures using pretrained weights and lightweight fine-tuning, recovering performance and improving efficiency.
transformers linear recurrent modeling linearization large language models LLMs gated recurrent models Liger key matrix weights gating mechanisms Low-Rank Adaptation LoRA Liger Attention intra-layer hybrid attention mechanism