ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL
Real-world robotic agents must act under partial observability and long
horizons, where key cues may appear long before they affect decision making.
However, most modern approaches rely solely on instantaneous information,
without incorporating insights from the past. Standard recurrent or transformer
models struggle with retaining and leveraging long-term dependencies: context
windows truncate history, while naive memory extensions fail under scale and
sparsity. We propose ELMUR (External Layer Memory with Update/Rewrite), a
transformer architecture with structured external memory. Each layer maintains
memory embeddings, interacts with them via bidirectional cross-attention, and
updates them through an Least Recently Used (LRU) memory module using
replacement or convex blending. ELMUR extends effective horizons up to 100,000
times beyond the attention window and achieves a 100% success rate on a
synthetic T-Maze task with corridors up to one million steps. In POPGym, it
outperforms baselines on more than half of the tasks. On MIKASA-Robo
sparse-reward manipulation tasks with visual observations, it nearly doubles
the performance of strong baselines. These results demonstrate that structured,
layer-local external memory offers a simple and scalable approach to decision
making under partial observability.
68ecf976b0aa732b88f31ffc
Egor Cherepanov
claimed_verified
2025-10-13T16:08:42.968Z
6668687caee0993c95b0eb81
/avatars/301fe1f395e0a129b1c9785868fa9858.svg
Egor Cherepanov
user
avanturist
68ecf976b0aa732b88f31ffd
Alexey K. Kovalev
68ecf976b0aa732b88f31ffe
Aleksandr I. Panov
ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL
Real-world robotic agents must act under partial observability and long
horizons, where key cues may appear long before they affect decision making.
However, most modern approaches rely solely on instantaneous information,
without incorporating insights from the past. Standard recurrent or transformer
models struggle with retaining and leveraging long-term dependencies: context
windows truncate history, while naive memory extensions fail under scale and
sparsity. We propose ELMUR (External Layer Memory with Update/Rewrite), a
transformer architecture with structured external memory. Each layer maintains
memory embeddings, interacts with them via bidirectional cross-attention, and
updates them through an Least Recently Used (LRU) memory module using
replacement or convex blending. ELMUR extends effective horizons up to 100,000
times beyond the attention window and achieves a 100% success rate on a
synthetic T-Maze task with corridors up to one million steps. In POPGym, it
outperforms baselines on more than half of the tasks. On MIKASA-Robo
sparse-reward manipulation tasks with visual observations, it nearly doubles
the performance of strong baselines. These results demonstrate that structured,
layer-local external memory offers a simple and scalable approach to decision
making under partial observability.
ELMUR, a transformer with structured external memory, enhances decision-making under partial observability by extending effective horizons and improving performance on various tasks.
recurrent models transformer models long-term dependencies context windows memory extensions ELMUR External Layer Memory bidirectional cross-attention Least Recently Used LRU memory module replacement convex blending T-Maze task POPGym MIKASA-Robo sparse-reward manipulation tasks visual observations