Block-State Transformer
State space models (SSMs) have shown impressive results on tasks that require
modeling long-range dependencies and efficiently scale to long sequences owing
to their subquadratic runtime complexity. Originally designed for continuous
signals, SSMs have shown superior performance on a plethora of tasks, in vision
and audio; however, SSMs still lag Transformer performance in Language Modeling
tasks. In this work, we propose a hybrid layer named Block-State Transformer
(BST), that internally combines an SSM sublayer for long-range
contextualization, and a Block Transformer sublayer for short-term
representation of sequences. We study three different, and completely
parallelizable, variants that integrate SSMs and block-wise attention. We show
that our model outperforms similar Transformer-based architectures on language
modeling perplexity and generalizes to longer sequences. In addition, the
Block-State Transformer demonstrates more than tenfold increase in speed at the
layer level compared to the Block-Recurrent Transformer when model
parallelization is employed.
648fb819cda0395a6f332943
Mahan Fathi
claimed_verified
2023-06-19T11:44:42.951Z
64903c2d9b1aaa0738286323
/avatars/9c4d0734b6f779d0f138951998f1d7b5.svg
Mahan Fathi
user
mahanfathi
648fb819cda0395a6f332944
Jonathan Pilault
648fb819cda0395a6f332945
Pierre-Luc Bacon
648fb819cda0395a6f332946
Christopher Pal
648fb819cda0395a6f332947
Orhan Firat
648fb819cda0395a6f332948
Ross Goroshin
Block-State Transformer
State space models (SSMs) have shown impressive results on tasks that require
modeling long-range dependencies and efficiently scale to long sequences owing
to their subquadratic runtime complexity. Originally designed for continuous
signals, SSMs have shown superior performance on a plethora of tasks, in vision
and audio; however, SSMs still lag Transformer performance in Language Modeling
tasks. In this work, we propose a hybrid layer named Block-State Transformer
(BST), that internally combines an SSM sublayer for long-range
contextualization, and a Block Transformer sublayer for short-term
representation of sequences. We study three different, and completely
parallelizable, variants that integrate SSMs and block-wise attention. We show
that our model outperforms similar Transformer-based architectures on language
modeling perplexity and generalizes to longer sequences. In addition, the
Block-State Transformer demonstrates more than tenfold increase in speed at the
layer level compared to the Block-Recurrent Transformer when model
parallelization is employed.
A hybrid model combining state space models and block-wise attention outperforms Transformer-based architectures in language modeling, offering faster processing speeds.
state space models Block-State Transformer long-range dependencies subquadratic runtime complexity Block Transformer block-wise attention language modeling perplexity model parallelization Block-Recurrent Transformer