Linear Transformers with Learnable Kernel Functions are Better
  In-Context Models
Advancing the frontier of subquadratic architectures for Language Models
(LMs) is crucial in the rapidly evolving field of natural language processing.
Current innovations, including State Space Models, were initially celebrated
for surpassing Transformer performance on language modeling tasks. However,
these models have revealed deficiencies in essential In-Context Learning
capabilities - a domain where the Transformer traditionally shines. The Based
model emerged as a hybrid solution, blending a Linear Transformer with a kernel
inspired by the Taylor expansion of exponential functions, augmented by
convolutional networks. Mirroring the Transformer's in-context adeptness, it
became a strong contender in the field. In our work, we present a singular,
elegant alteration to the Based kernel that amplifies its In-Context Learning
abilities evaluated with the Multi-Query Associative Recall task and overall
language modeling process, as demonstrated on the Pile dataset.
65d2e27c8120c6edb8230737
Yaroslav Aksenov
claimed_verified
2024-02-19T17:05:22.159Z
63ed5676684767daecac6f8a
/avatars/d0e4a715f9c3fb6d74c183bab751ec35.svg
Yaroslav Aksenov
user
yaraksen
65d2e27c8120c6edb8230738
Nikita Balagansky
admin_assigned
2024-02-19T15:13:57.032Z
60b364e7f88532cd79eaff7b
https://cdn-avatars.huggingface.co/v1/production/uploads/1654185363389-60b364e7f88532cd79eaff7b.jpeg
Nikita Balagansky
user
elephantmipt
65d2e27c8120c6edb8230739
Sofia Maria Lo Cicero Vaina
claimed_verified
2024-02-20T08:39:04.278Z
65d384f2d717f5ae95dd9099
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KPyo3z_hjEs6qlSVRnNH9.jpeg
Sofia Lo Cicero
user
2son1a
65d2e27c8120c6edb823073a
Boris Shaposhnikov
claimed_verified
2024-04-16T09:04:49.315Z
637dd11dcbad6e62a5e39743
https://cdn-avatars.huggingface.co/v1/production/uploads/637dd11dcbad6e62a5e39743/DG3rM8cy8inqbCoG4qizO.jpeg
Boris Shaposhnikov
user
borisshapa
65d2e27c8120c6edb823073b
Alexey Gorbatovski
claimed_verified
2024-04-16T08:59:48.728Z
62897fce5d9e25c10e4f319d
https://cdn-avatars.huggingface.co/v1/production/uploads/62897fce5d9e25c10e4f319d/bMlfAyzkNNZlkQ5mCW6Vc.jpeg
Alexey Gorbatovski
user
Myashka
65d2e27c8120c6edb823073c
Daniil Gavrilov
admin_assigned
2024-02-19T15:15:23.473Z
62a9c8edc19f92ae443ab37f
https://cdn-avatars.huggingface.co/v1/production/uploads/1669110208492-62a9c8edc19f92ae443ab37f.png
Daniil Gavrilov
user
kefirski
Linear Transformers with Learnable Kernel Functions are Better
  In-Context Models
Advancing the frontier of subquadratic architectures for Language Models
(LMs) is crucial in the rapidly evolving field of natural language processing.
Current innovations, including State Space Models, were initially celebrated
for surpassing Transformer performance on language modeling tasks. However,
these models have revealed deficiencies in essential In-Context Learning
capabilities - a domain where the Transformer traditionally shines. The Based
model emerged as a hybrid solution, blending a Linear Transformer with a kernel
inspired by the Taylor expansion of exponential functions, augmented by
convolutional networks. Mirroring the Transformer's in-context adeptness, it
became a strong contender in the field. In our work, we present a singular,
elegant alteration to the Based kernel that amplifies its In-Context Learning
abilities evaluated with the Multi-Query Associative Recall task and overall
language modeling process, as demonstrated on the Pile dataset.
A modification to the Based model kernel enhances its in-context learning capabilities, outperforming existing subquadratic architectures in language modeling tasks on the Pile dataset.
State Space Models Transformer In-Context Learning Based model Linear Transformer Taylor expansion convolutional networks Multi-Query Associative Recall Pile dataset