Complexity of Symbolic Representation in Working Memory of Transformer
  Correlates with the Complexity of a Task
Even though Transformers are extensively used for Natural Language Processing
tasks, especially for machine translation, they lack an explicit memory to
store key concepts of processed texts. This paper explores the properties of
the content of symbolic working memory added to the Transformer model decoder.
Such working memory enhances the quality of model predictions in machine
translation task and works as a neural-symbolic representation of information
that is important for the model to make correct translations. The study of
memory content revealed that translated text keywords are stored in the working
memory, pointing to the relevance of memory content to the processed text.
Also, the diversity of tokens and parts of speech stored in memory correlates
with the complexity of the corpora for machine translation task.
66753855e2494e5447dd7a5e
Alsu Sagirova
extracted_confirmed
2024-06-21T08:23:35.153Z
65c0db0fbda79a18292dfbb7
/avatars/1201b8282664c2d8c18beaba2396c03b.svg
Alsu Sagirova
user
alsu-sagirova
66753855e2494e5447dd7a5f
Mikhail Burtsev
admin_assigned
2024-06-24T14:01:43.806Z
639c6e978a34ed9a404c6a7b
/avatars/c98ca8c9f9ed8509c2f1bb6aa994fd57.svg
MIKHAIL BURTSEV
user
mbur
Complexity of Symbolic Representation in Working Memory of Transformer
  Correlates with the Complexity of a Task
Even though Transformers are extensively used for Natural Language Processing
tasks, especially for machine translation, they lack an explicit memory to
store key concepts of processed texts. This paper explores the properties of
the content of symbolic working memory added to the Transformer model decoder.
Such working memory enhances the quality of model predictions in machine
translation task and works as a neural-symbolic representation of information
that is important for the model to make correct translations. The study of
memory content revealed that translated text keywords are stored in the working
memory, pointing to the relevance of memory content to the processed text.
Also, the diversity of tokens and parts of speech stored in memory correlates
with the complexity of the corpora for machine translation task.
The addition of symbolic working memory to Transformer decoders improves machine translation by storing key concepts, with memory content correlating to text complexity.
Transformers machine translation symbolic working memory neural-symbolic representation translated text keywords token diversity parts of speech corpus complexity