NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged
  Information Guidance
Learning navigation in dynamic open-world environments is an important yet
challenging skill for robots. Most previous methods rely on precise
localization and mapping or learn from expensive real-world demonstrations. In
this paper, we propose the Navigation Diffusion Policy (NavDP), an end-to-end
framework trained solely in simulation and can zero-shot transfer to different
embodiments in diverse real-world environments. The key ingredient of NavDP's
network is the combination of diffusion-based trajectory generation and a
critic function for trajectory selection, which are conditioned on only local
observation tokens encoded from a shared policy transformer. Given the
privileged information of the global environment in simulation, we scale up the
demonstrations of good quality to train the diffusion policy and formulate the
critic value function targets with contrastive negative samples. Our
demonstration generation approach achieves about 2,500 trajectories/GPU per
day, 20times more efficient than real-world data collection, and results in
a large-scale navigation dataset with 363.2km trajectories across 1244 scenes.
Trained with this simulation dataset, NavDP achieves state-of-the-art
performance and consistently outstanding generalization capability on
quadruped, wheeled, and humanoid robots in diverse indoor and outdoor
environments. In addition, we present a preliminary attempt at using Gaussian
Splatting to make in-domain real-to-sim fine-tuning to further bridge the
sim-to-real gap. Experiments show that adding such real-to-sim data can improve
the success rate by 30\% without hurting its generalization capability.
682451c487e04e8c4ee5d13b
Wenzhe Cai
admin_assigned
2025-05-14T10:12:04.232Z
66a347adb839c8994e6cb641
/avatars/efdeff32628b6c531109e047b45b2627.svg
Wenzhe Cai
user
WadeCai
682451c487e04e8c4ee5d13c
Jiaqi Peng
682451c487e04e8c4ee5d13d
Yuqiang Yang
admin_assigned
2025-05-14T10:12:17.389Z
670bbd8541e624a441f76306
/avatars/b606cabd30f1374b1ffa82ff1b7e9ae6.svg
yuqiang yang
user
fulifuli666
682451c487e04e8c4ee5d13e
Yujian Zhang
682451c487e04e8c4ee5d13f
Meng Wei
682451c487e04e8c4ee5d140
Hanqing Wang
claimed_verified
2025-07-16T15:07:41.501Z
65d9f09bbcd15bc5cb255fed
/avatars/4ae7c5366a6454e542bf3b3ddefa9c7e.svg
hanqingwang
user
hanqing94
682451c487e04e8c4ee5d141
Yilun Chen
682451c487e04e8c4ee5d142
Tai Wang
682451c487e04e8c4ee5d143
Jiangmiao Pang
admin_assigned
2025-05-14T10:13:19.562Z
65783ee6ee33d547aecc3ffc
https://cdn-avatars.huggingface.co/v1/production/uploads/65783ee6ee33d547aecc3ffc/lWZX88c-0dCsN-yB9Jhlf.jpeg
Jiangmiao Pang
user
Jiangmiao
NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged
  Information Guidance
Learning navigation in dynamic open-world environments is an important yet
challenging skill for robots. Most previous methods rely on precise
localization and mapping or learn from expensive real-world demonstrations. In
this paper, we propose the Navigation Diffusion Policy (NavDP), an end-to-end
framework trained solely in simulation and can zero-shot transfer to different
embodiments in diverse real-world environments. The key ingredient of NavDP's
network is the combination of diffusion-based trajectory generation and a
critic function for trajectory selection, which are conditioned on only local
observation tokens encoded from a shared policy transformer. Given the
privileged information of the global environment in simulation, we scale up the
demonstrations of good quality to train the diffusion policy and formulate the
critic value function targets with contrastive negative samples. Our
demonstration generation approach achieves about 2,500 trajectories/GPU per
day, 20times more efficient than real-world data collection, and results in
a large-scale navigation dataset with 363.2km trajectories across 1244 scenes.
Trained with this simulation dataset, NavDP achieves state-of-the-art
performance and consistently outstanding generalization capability on
quadruped, wheeled, and humanoid robots in diverse indoor and outdoor
environments. In addition, we present a preliminary attempt at using Gaussian
Splatting to make in-domain real-to-sim fine-tuning to further bridge the
sim-to-real gap. Experiments show that adding such real-to-sim data can improve
the success rate by 30\% without hurting its generalization capability.
NavDP, an end-to-end diffusion policy framework trained in simulation, achieves state-of-the-art performance and generalization in diverse real-world environments through efficient data generation and real-to-sim fine-tuning using Gaussian Splatting.
navigation diffusion policy NavDP diffusion-based trajectory generation critic function trajectory selection local observation tokens policy transformer global environment contrastive negative samples Gaussian Splatting sim-to-real gap