Learning to Manipulate Anywhere: A Visual Generalizable Framework For
  Reinforcement Learning
Can we endow visuomotor robots with generalization capabilities to operate in
diverse open-world scenarios? In this paper, we propose Maniwhere, a
generalizable framework tailored for visual reinforcement learning, enabling
the trained robot policies to generalize across a combination of multiple
visual disturbance types. Specifically, we introduce a multi-view
representation learning approach fused with Spatial Transformer Network (STN)
module to capture shared semantic information and correspondences among
different viewpoints. In addition, we employ a curriculum-based randomization
and augmentation approach to stabilize the RL training process and strengthen
the visual generalization ability. To exhibit the effectiveness of Maniwhere,
we meticulously design 8 tasks encompassing articulate objects, bi-manual, and
dexterous hand manipulation tasks, demonstrating Maniwhere's strong visual
generalization and sim2real transfer abilities across 3 hardware platforms. Our
experiments show that Maniwhere significantly outperforms existing
state-of-the-art methods. Videos are provided at
https://gemcollector.github.io/maniwhere/.
66a1c2d437b81357eff321f9
Zhecheng Yuan
claimed_verified
2024-07-25T07:38:24.557Z
64a0dd823886663ff8455f7a
/avatars/a8560f9c9de59cbab4ef5ed9494e4051.svg
Zhecheng Yuan
user
gemcollector
66a1c2d437b81357eff321fa
Tianming Wei
66a1c2d437b81357eff321fb
Shuiqi Cheng
admin_assigned
2024-07-25T12:25:15.728Z
65f493d1a8d2ff8f8f94689d
/avatars/4f01775f79be964d4ee4a6d82b3b5943.svg
shuiqi cheng
user
shuiqicheng
66a1c2d437b81357eff321fc
Gu Zhang
admin_assigned
2024-07-25T12:25:33.952Z
65e95effe19214e9d13d7a2a
/avatars/35ae40579ac53d7b598957be9c8b4710.svg
Gu Zhang
user
GuZhang
66a1c2d437b81357eff321fd
Yuanpei Chen
admin_assigned
2024-07-25T12:25:42.807Z
6393a8af84c565d2c3419b7c
/avatars/f2a237a58dd0a25ef5c1a98e60acbb5c.svg
chen
user
yuanpei
66a1c2d437b81357eff321fe
Huazhe Xu
admin_assigned
2024-07-25T12:25:48.615Z
649c130f92525b3ac9fbcffe
/avatars/b60e4d6803f7e4ef42200ddcb9e5bac1.svg
Huazhe Xu
user
Huazhe-Xu
Learning to Manipulate Anywhere: A Visual Generalizable Framework For
  Reinforcement Learning
Can we endow visuomotor robots with generalization capabilities to operate in
diverse open-world scenarios? In this paper, we propose Maniwhere, a
generalizable framework tailored for visual reinforcement learning, enabling
the trained robot policies to generalize across a combination of multiple
visual disturbance types. Specifically, we introduce a multi-view
representation learning approach fused with Spatial Transformer Network (STN)
module to capture shared semantic information and correspondences among
different viewpoints. In addition, we employ a curriculum-based randomization
and augmentation approach to stabilize the RL training process and strengthen
the visual generalization ability. To exhibit the effectiveness of Maniwhere,
we meticulously design 8 tasks encompassing articulate objects, bi-manual, and
dexterous hand manipulation tasks, demonstrating Maniwhere's strong visual
generalization and sim2real transfer abilities across 3 hardware platforms. Our
experiments show that Maniwhere significantly outperforms existing
state-of-the-art methods. Videos are provided at
https://gemcollector.github.io/maniwhere/.
Maniwhere, a visual reinforcement learning framework, enables robotic policies to generalize across varied visual disturbances using multi-view representation and Spatial Transformer Network, demonstrating superior performance in sim2real transfer.
multi-view representation Spatial Transformer Network (STN) curriculum-based randomization augmentation visual reinforcement learning visual generalization sim2real transfer articulate objects bi-manual manipulation dexterous hand manipulation