StateX: Enhancing RNN Recall via Post-training State Expansion
While Transformer-based models have demonstrated remarkable language modeling
performance, their high complexities result in high costs when processing long
contexts. In contrast, recurrent neural networks (RNNs) such as linear
attention and state space models have gained popularity due to their constant
per-token complexities. However, these recurrent models struggle with tasks
that require accurate recall of contextual information from long contexts,
because all contextual information is compressed into a constant-size recurrent
state. Previous works have shown that recall ability is positively correlated
with the recurrent state size, yet directly training RNNs with larger recurrent
states results in high training costs. In this paper, we introduce StateX, a
training pipeline for efficiently expanding the states of pre-trained RNNs
through post-training. For two popular classes of RNNs, linear attention and
state space models, we design post-training architectural modifications to
scale up the state size with no or negligible increase in model parameters.
Experiments on models up to 1.3B parameters demonstrate that StateX efficiently
enhances the recall and in-context learning ability of RNNs without incurring
high post-training costs or compromising other capabilities.
68da035b0177a6054b013b85
Xingyu Shen
68da035b0177a6054b013b86
Yingfa Chen
claimed_verified
2025-09-29T05:57:43.162Z
6144e4667f2544bb450787b2
https://cdn-avatars.huggingface.co/v1/production/uploads/6144e4667f2544bb450787b2/5wSDDqJbI4TGtBLP9IvjY.png
Yingfa Chen
user
chen-yingfa
68da035b0177a6054b013b87
Zhen Leng Thai
68da035b0177a6054b013b88
Xu Han
68da035b0177a6054b013b89
Zhiyuan Liu
68da035b0177a6054b013b8a
Maosong Sun
StateX: Enhancing RNN Recall via Post-training State Expansion
While Transformer-based models have demonstrated remarkable language modeling
performance, their high complexities result in high costs when processing long
contexts. In contrast, recurrent neural networks (RNNs) such as linear
attention and state space models have gained popularity due to their constant
per-token complexities. However, these recurrent models struggle with tasks
that require accurate recall of contextual information from long contexts,
because all contextual information is compressed into a constant-size recurrent
state. Previous works have shown that recall ability is positively correlated
with the recurrent state size, yet directly training RNNs with larger recurrent
states results in high training costs. In this paper, we introduce StateX, a
training pipeline for efficiently expanding the states of pre-trained RNNs
through post-training. For two popular classes of RNNs, linear attention and
state space models, we design post-training architectural modifications to
scale up the state size with no or negligible increase in model parameters.
Experiments on models up to 1.3B parameters demonstrate that StateX efficiently
enhances the recall and in-context learning ability of RNNs without incurring
high post-training costs or compromising other capabilities.
StateX is a post-training pipeline that expands the state size of pre-trained RNNs, enhancing recall and in-context learning without significant additional costs.
Transformer-based models language modeling recurrent neural networks RNNs linear attention state space models recurrent state StateX post-training in-context learning