Unraveling the Complexity of Memory in RL Agents: an Approach for
  Classification and Evaluation
The incorporation of memory into agents is essential for numerous tasks
within the domain of Reinforcement Learning (RL). In particular, memory is
paramount for tasks that require the utilization of past information,
adaptation to novel environments, and improved sample efficiency. However, the
term ``memory'' encompasses a wide range of concepts, which, coupled with the
lack of a unified methodology for validating an agent's memory, leads to
erroneous judgments about agents' memory capabilities and prevents objective
comparison with other memory-enhanced agents. This paper aims to streamline the
concept of memory in RL by providing practical precise definitions of agent
memory types, such as long-term versus short-term memory and declarative versus
procedural memory, inspired by cognitive science. Using these definitions, we
categorize different classes of agent memory, propose a robust experimental
methodology for evaluating the memory capabilities of RL agents, and
standardize evaluations. Furthermore, we empirically demonstrate the importance
of adhering to the proposed methodology when evaluating different types of
agent memory by conducting experiments with different RL agents and what its
violation leads to.
67583ace95bb6a9c895df272
Egor Cherepanov
claimed_verified
2024-12-10T13:06:34.114Z
6668687caee0993c95b0eb81
/avatars/301fe1f395e0a129b1c9785868fa9858.svg
Egor Cherepanov
user
avanturist
67583ace95bb6a9c895df273
Nikita Kachaev
claimed_verified
2024-12-10T15:15:21.788Z
64705ef84be5cf1f3348e283
/avatars/915875e7c4118098ab460831d5e8ef0e.svg
Nikita
user
tttonyalpha
67583ace95bb6a9c895df274
Artem Zholus
admin_assigned
2024-12-10T13:07:54.862Z
6267cdd3f91d1c1633c08bbf
/avatars/51d7961098f52f39ee406009a12982b8.svg
Artem Zholus
user
artemZholus
67583ace95bb6a9c895df275
Alexey K. Kovalev
admin_assigned
2024-12-10T13:07:49.350Z
64198f70ed725fef6442b37e
/avatars/580ab07a3067a9deb2977b0894226fe3.svg
Alexey Kovalev
user
AlexeyKov
67583ace95bb6a9c895df276
Aleksandr I. Panov
admin_assigned
2024-12-10T13:07:02.724Z
6633243f5ddb7702ad3ec216
/avatars/0abaf60f1a52e148c3f6fce57eb31eb4.svg
Aleksandr Panov
user
grafft
Unraveling the Complexity of Memory in RL Agents: an Approach for
  Classification and Evaluation
The incorporation of memory into agents is essential for numerous tasks
within the domain of Reinforcement Learning (RL). In particular, memory is
paramount for tasks that require the utilization of past information,
adaptation to novel environments, and improved sample efficiency. However, the
term ``memory'' encompasses a wide range of concepts, which, coupled with the
lack of a unified methodology for validating an agent's memory, leads to
erroneous judgments about agents' memory capabilities and prevents objective
comparison with other memory-enhanced agents. This paper aims to streamline the
concept of memory in RL by providing practical precise definitions of agent
memory types, such as long-term versus short-term memory and declarative versus
procedural memory, inspired by cognitive science. Using these definitions, we
categorize different classes of agent memory, propose a robust experimental
methodology for evaluating the memory capabilities of RL agents, and
standardize evaluations. Furthermore, we empirically demonstrate the importance
of adhering to the proposed methodology when evaluating different types of
agent memory by conducting experiments with different RL agents and what its
violation leads to.
The paper provides a framework for understanding and evaluating different types of memory in reinforcement learning agents, using cognitive science-inspired definitions and a standardized experimental methodology.
Reinforcement Learning RL long-term memory short-term memory declarative memory procedural memory cognitive science memory capabilities evaluation methodology sample efficiency