HERMES: Human-to-Robot Embodied Learning from Multi-Source Motion Data
  for Mobile Dexterous Manipulation
Leveraging human motion data to impart robots with versatile manipulation
skills has emerged as a promising paradigm in robotic manipulation.
Nevertheless, translating multi-source human hand motions into feasible robot
behaviors remains challenging, particularly for robots equipped with
multi-fingered dexterous hands characterized by complex, high-dimensional
action spaces. Moreover, existing approaches often struggle to produce policies
capable of adapting to diverse environmental conditions. In this paper, we
introduce HERMES, a human-to-robot learning framework for mobile bimanual
dexterous manipulation. First, HERMES formulates a unified reinforcement
learning approach capable of seamlessly transforming heterogeneous human hand
motions from multiple sources into physically plausible robotic behaviors.
Subsequently, to mitigate the sim2real gap, we devise an end-to-end, depth
image-based sim2real transfer method for improved generalization to real-world
scenarios. Furthermore, to enable autonomous operation in varied and
unstructured environments, we augment the navigation foundation model with a
closed-loop Perspective-n-Point (PnP) localization mechanism, ensuring precise
alignment of visual goals and effectively bridging autonomous navigation and
dexterous manipulation. Extensive experimental results demonstrate that HERMES
consistently exhibits generalizable behaviors across diverse, in-the-wild
scenarios, successfully performing numerous complex mobile bimanual dexterous
manipulation tasks. Project Page:https://gemcollector.github.io/HERMES/.
68b523c6851c6e7b001ecab0
Zhecheng Yuan
68b523c6851c6e7b001ecab1
Tianming Wei
68b523c6851c6e7b001ecab2
Langzhe Gu
68b523c6851c6e7b001ecab3
Pu Hua
68b523c6851c6e7b001ecab4
Tianhai Liang
68b523c6851c6e7b001ecab5
Yuanpei Chen
68b523c6851c6e7b001ecab6
Huazhe Xu
HERMES: Human-to-Robot Embodied Learning from Multi-Source Motion Data
  for Mobile Dexterous Manipulation
Leveraging human motion data to impart robots with versatile manipulation
skills has emerged as a promising paradigm in robotic manipulation.
Nevertheless, translating multi-source human hand motions into feasible robot
behaviors remains challenging, particularly for robots equipped with
multi-fingered dexterous hands characterized by complex, high-dimensional
action spaces. Moreover, existing approaches often struggle to produce policies
capable of adapting to diverse environmental conditions. In this paper, we
introduce HERMES, a human-to-robot learning framework for mobile bimanual
dexterous manipulation. First, HERMES formulates a unified reinforcement
learning approach capable of seamlessly transforming heterogeneous human hand
motions from multiple sources into physically plausible robotic behaviors.
Subsequently, to mitigate the sim2real gap, we devise an end-to-end, depth
image-based sim2real transfer method for improved generalization to real-world
scenarios. Furthermore, to enable autonomous operation in varied and
unstructured environments, we augment the navigation foundation model with a
closed-loop Perspective-n-Point (PnP) localization mechanism, ensuring precise
alignment of visual goals and effectively bridging autonomous navigation and
dexterous manipulation. Extensive experimental results demonstrate that HERMES
consistently exhibits generalizable behaviors across diverse, in-the-wild
scenarios, successfully performing numerous complex mobile bimanual dexterous
manipulation tasks. Project Page:https://gemcollector.github.io/HERMES/.
HERMES is a human-to-robot learning framework that translates human hand motions into robotic behaviors, using reinforcement learning and sim2real transfer for versatile manipulation in diverse environments.
reinforcement learning sim2real transfer Perspective-n-Point (PnP) localization mobile bimanual dexterous manipulation