Associative Recurrent Memory Transformer
This paper addresses the challenge of creating a neural architecture for very
long sequences that requires constant time for processing new information at
each time step. Our approach, Associative Recurrent Memory Transformer (ARMT),
is based on transformer self-attention for local context and segment-level
recurrence for storage of task specific information distributed over a long
context. We demonstrate that ARMT outperfors existing alternatives in
associative retrieval tasks and sets a new performance record in the recent
BABILong multi-task long-context benchmark by answering single-fact questions
over 50 million tokens with an accuracy of 79.9%. The source code for training
and evaluation is available on github.
668cee02f8e6657dff5baeb7
Ivan Rodkin
claimed_verified
2024-07-09T21:09:12.111Z
63c1ac8cc58fcfeac186bda2
https://cdn-avatars.huggingface.co/v1/production/uploads/63c1ac8cc58fcfeac186bda2/MajMhLgsvCBN81tmO9PZc.jpeg
Ivan Rodkin
user
irodkin
668cee02f8e6657dff5baeb8
Yuri Kuratov
claimed_verified
2024-07-09T08:17:45.419Z
618b9540682ec1c38327e586
https://cdn-avatars.huggingface.co/v1/production/uploads/618b9540682ec1c38327e586/v_ZBkfh8O9Zh6C2YQpuBX.jpeg
Yury Kuratov
user
yurakuratov
668cee02f8e6657dff5baeb9
Aydar Bulatov
claimed_verified
2025-02-21T14:43:30.781Z
64c8b321cb2f1bf0e7c0f54b
https://cdn-avatars.huggingface.co/v1/production/uploads/64c8b321cb2f1bf0e7c0f54b/JflXxMVnG9I0IB5YNyhXF.jpeg
Aydar Bulatov
user
booydar
668cee02f8e6657dff5baeba
Mikhail Burtsev
claimed_verified
2024-07-18T09:08:18.446Z
639c6e978a34ed9a404c6a7b
/avatars/c98ca8c9f9ed8509c2f1bb6aa994fd57.svg
MIKHAIL BURTSEV
user
mbur
Associative Recurrent Memory Transformer
This paper addresses the challenge of creating a neural architecture for very
long sequences that requires constant time for processing new information at
each time step. Our approach, Associative Recurrent Memory Transformer (ARMT),
is based on transformer self-attention for local context and segment-level
recurrence for storage of task specific information distributed over a long
context. We demonstrate that ARMT outperfors existing alternatives in
associative retrieval tasks and sets a new performance record in the recent
BABILong multi-task long-context benchmark by answering single-fact questions
over 50 million tokens with an accuracy of 79.9%. The source code for training
and evaluation is available on github.
The Associative Recurrent Memory Transformer effectively handles long sequences by combining transformer self-attention with segment-level recurrence, achieving high accuracy in long-context tasks like BABILong.
transformer self-attention segment-level recurrence Associative Recurrent Memory Transformer BABILong benchmark