Cached Transformers: Improving Transformers with Differentiable Memory
  Cache
This work introduces a new Transformer model called Cached Transformer, which
uses Gated Recurrent Cached (GRC) attention to extend the self-attention
mechanism with a differentiable memory cache of tokens. GRC attention enables
attending to both past and current tokens, increasing the receptive field of
attention and allowing for exploring long-range dependencies. By utilizing a
recurrent gating unit to continuously update the cache, our model achieves
significant advancements in six language and vision tasks, including
language modeling, machine translation, ListOPs, image classification, object
detection, and instance segmentation. Furthermore, our approach surpasses
previous memory-based techniques in tasks such as language modeling and
displays the ability to be applied to a broader range of situations.
6583b44583a9e1460c68f5e7
Zhaoyang Zhang
claimed_verified
2023-12-21T14:05:47.894Z
658409ceca19ccf6d9989add
/avatars/3ac1dbd25e52435185babdeb3da28875.svg
Zhaoyang Zhang
user
ZyZcuhk
6583b44583a9e1460c68f5e8
Wenqi Shao
admin_assigned
2023-12-21T08:52:08.198Z
64b3fd42eec33e27dcc4c941
/avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
Wenqi Shao
user
wqshao126
6583b44583a9e1460c68f5e9
Yixiao Ge
admin_assigned
2023-12-21T08:52:14.862Z
640e9762b03f4cd29f58d982
/avatars/81da37d628163fe3e094b247c7c3a3b5.svg
Yixiao Ge
user
yxgeee
6583b44583a9e1460c68f5ea
Xiaogang Wang
6583b44583a9e1460c68f5eb
Jinwei Gu
admin_assigned
2023-12-21T08:52:30.811Z
647e8118770c299e56fc2bc8
/avatars/adf80f3473dda42450148789ae5c208f.svg
Jinwei Gu
user
jwgu
6583b44583a9e1460c68f5ec
Ping Luo
Cached Transformers: Improving Transformers with Differentiable Memory
  Cache
This work introduces a new Transformer model called Cached Transformer, which
uses Gated Recurrent Cached (GRC) attention to extend the self-attention
mechanism with a differentiable memory cache of tokens. GRC attention enables
attending to both past and current tokens, increasing the receptive field of
attention and allowing for exploring long-range dependencies. By utilizing a
recurrent gating unit to continuously update the cache, our model achieves
significant advancements in six language and vision tasks, including
language modeling, machine translation, ListOPs, image classification, object
detection, and instance segmentation. Furthermore, our approach surpasses
previous memory-based techniques in tasks such as language modeling and
displays the ability to be applied to a broader range of situations.
A Cached Transformer model with Gated Recurrent Cached attention achieves significant advancements in language and vision tasks by extending self-attention with a differentiable memory cache.
Transformer Gated Recurrent Cached (GRC) attention self-attention differentiable memory cache recurrent gating unit language modeling machine translation ListOPs image classification object detection instance segmentation