Mem4Nav: Boosting Vision-and-Language Navigation in Urban Environments
  with a Hierarchical Spatial-Cognition Long-Short Memory System
Vision-and-Language Navigation (VLN) in large-scale urban environments
requires embodied agents to ground linguistic instructions in complex scenes
and recall relevant experiences over extended time horizons. Prior modular
pipelines offer interpretability but lack unified memory, while end-to-end
(M)LLM agents excel at fusing vision and language yet remain constrained by
fixed context windows and implicit spatial reasoning. We introduce
Mem4Nav, a hierarchical spatial-cognition long-short memory system
that can augment any VLN backbone. Mem4Nav fuses a sparse octree for
fine-grained voxel indexing with a semantic topology graph for high-level
landmark connectivity, storing both in trainable memory tokens embedded via a
reversible Transformer. Long-term memory (LTM) compresses and retains
historical observations at both octree and graph nodes, while short-term memory
(STM) caches recent multimodal entries in relative coordinates for real-time
obstacle avoidance and local planning. At each step, STM retrieval sharply
prunes dynamic context, and, when deeper history is needed, LTM tokens are
decoded losslessly to reconstruct past embeddings. Evaluated on Touchdown and
Map2Seq across three backbones (modular, state-of-the-art VLN with prompt-based
LLM, and state-of-the-art VLN with strided-attention MLLM), Mem4Nav yields 7-13
pp gains in Task Completion, sufficient SPD reduction, and >10 pp nDTW
improvement. Ablations confirm the indispensability of both the hierarchical
map and dual memory modules. Our codes are open-sourced via
https://github.com/tsinghua-fib-lab/Mem4Nav.
685bd821aafa002dfd2bdf09
Lixuan He
685bd821aafa002dfd2bdf0a
Haoyu Dong
685bd821aafa002dfd2bdf0b
Zhenxing Chen
685bd821aafa002dfd2bdf0c
Yangcheng Yu
685bd821aafa002dfd2bdf0d
Jie Feng
claimed_verified
2025-06-25T12:56:19.744Z
6465d3bd63e7e09dd02e95c3
/avatars/b2798bd5f8368f956bf7fab79d9432f0.svg
Jie Feng
user
JJ-TMT
685bd821aafa002dfd2bdf0e
Yong Li
Mem4Nav: Boosting Vision-and-Language Navigation in Urban Environments
  with a Hierarchical Spatial-Cognition Long-Short Memory System
Vision-and-Language Navigation (VLN) in large-scale urban environments
requires embodied agents to ground linguistic instructions in complex scenes
and recall relevant experiences over extended time horizons. Prior modular
pipelines offer interpretability but lack unified memory, while end-to-end
(M)LLM agents excel at fusing vision and language yet remain constrained by
fixed context windows and implicit spatial reasoning. We introduce
Mem4Nav, a hierarchical spatial-cognition long-short memory system
that can augment any VLN backbone. Mem4Nav fuses a sparse octree for
fine-grained voxel indexing with a semantic topology graph for high-level
landmark connectivity, storing both in trainable memory tokens embedded via a
reversible Transformer. Long-term memory (LTM) compresses and retains
historical observations at both octree and graph nodes, while short-term memory
(STM) caches recent multimodal entries in relative coordinates for real-time
obstacle avoidance and local planning. At each step, STM retrieval sharply
prunes dynamic context, and, when deeper history is needed, LTM tokens are
decoded losslessly to reconstruct past embeddings. Evaluated on Touchdown and
Map2Seq across three backbones (modular, state-of-the-art VLN with prompt-based
LLM, and state-of-the-art VLN with strided-attention MLLM), Mem4Nav yields 7-13
pp gains in Task Completion, sufficient SPD reduction, and >10 pp nDTW
improvement. Ablations confirm the indispensability of both the hierarchical
map and dual memory modules. Our codes are open-sourced via
https://github.com/tsinghua-fib-lab/Mem4Nav.
Mem4Nav enhances Vision-and-Language Navigation by integrating a hierarchical spatial-cognition system with dual-memory modules using a reversible Transformer for improved task completion, speed, and detour detection.
sparse octree semantic topology graph trainable memory tokens reversible Transformer long-term memory short-term memory Touchdown Map2Seq modular VLN prompt-based LLM strided-attention MLLM hierarchical map dual memory modules