WildLMa: Long Horizon Loco-Manipulation in the Wild
`In-the-wild' mobile manipulation aims to deploy robots in diverse real-world
environments, which requires the robot to (1) have skills that generalize
across object configurations; (2) be capable of long-horizon task execution in
diverse environments; and (3) perform complex manipulation beyond
pick-and-place. Quadruped robots with manipulators hold promise for extending
the workspace and enabling robust locomotion, but existing results do not
investigate such a capability. This paper proposes WildLMa with three
components to address these issues: (1) adaptation of learned low-level
controller for VR-enabled whole-body teleoperation and traversability; (2)
WildLMa-Skill -- a library of generalizable visuomotor skills acquired via
imitation learning or heuristics and (3) WildLMa-Planner -- an interface of
learned skills that allow LLM planners to coordinate skills for long-horizon
tasks. We demonstrate the importance of high-quality training data by achieving
higher grasping success rate over existing RL baselines using only tens of
demonstrations. WildLMa exploits CLIP for language-conditioned imitation
learning that empirically generalizes to objects unseen in training
demonstrations. Besides extensive quantitative evaluation, we qualitatively
demonstrate practical robot applications, such as cleaning up trash in
university hallways or outdoor terrains, operating articulated objects, and
rearranging items on a bookshelf.
674413c5b5259d64b54af07b
Ri-Zhao Qiu
claimed_verified
2024-11-25T21:38:57.101Z
64eced8b6343a356327f5b45
/avatars/cf9de701a6fa893c7989773c4875c254.svg
Roger Qiu
user
RogerQi
674413c5b5259d64b54af07c
Yuchen Song
claimed_verified
2024-11-25T21:18:12.708Z
65482cb95f6c84d8f4c2267e
/avatars/c047bbb89db5bafe12f51e88a6f6dd0c.svg
Yuchen Song
user
YuchenEasonSong
674413c5b5259d64b54af07d
Xuanbin Peng
claimed_verified
2024-11-25T20:39:15.857Z
66f717cce5d2d88a93fe33a5
/avatars/dc9824ba08db7be050df6ba97e1da84c.svg
Xuanbin Peng
user
xuanbin-peng
674413c5b5259d64b54af07e
Sai Aneesh Suryadevara
674413c5b5259d64b54af07f
Ge Yang
admin_assigned
2024-11-25T21:07:41.294Z
5f03ee98ff69163f6f59e9e2
https://cdn-avatars.huggingface.co/v1/production/uploads/1594093169905-noauth.jpeg
Ge Yang
user
geyang
674413c5b5259d64b54af080
Minghuan Liu
admin_assigned
2024-11-25T21:01:08.405Z
630c8f0c8e3ff0c72327140a
/avatars/04a5f2effd0ff25d9fb531b195fd8846.svg
Minghuan Liu
user
ericonaldo
674413c5b5259d64b54af081
Mazeyu Ji
admin_assigned
2024-11-25T21:07:19.940Z
657eba2cf4f72f2c4c3d3862
/avatars/51624276469091758c2347147bb5cc4b.svg
Mazeyu Ji
user
jimazeyu
674413c5b5259d64b54af082
Chengzhe Jia
674413c5b5259d64b54af083
Ruihan Yang
674413c5b5259d64b54af084
Xueyan Zou
admin_assigned
2024-11-25T21:06:43.964Z
62520988818a5dc29ab91d6f
https://cdn-avatars.huggingface.co/v1/production/uploads/1671670066375-62520988818a5dc29ab91d6f.png
Xueyan Zou
user
xueyanz
674413c5b5259d64b54af085
Xiaolong Wang
WildLMa: Long Horizon Loco-Manipulation in the Wild
`In-the-wild' mobile manipulation aims to deploy robots in diverse real-world
environments, which requires the robot to (1) have skills that generalize
across object configurations; (2) be capable of long-horizon task execution in
diverse environments; and (3) perform complex manipulation beyond
pick-and-place. Quadruped robots with manipulators hold promise for extending
the workspace and enabling robust locomotion, but existing results do not
investigate such a capability. This paper proposes WildLMa with three
components to address these issues: (1) adaptation of learned low-level
controller for VR-enabled whole-body teleoperation and traversability; (2)
WildLMa-Skill -- a library of generalizable visuomotor skills acquired via
imitation learning or heuristics and (3) WildLMa-Planner -- an interface of
learned skills that allow LLM planners to coordinate skills for long-horizon
tasks. We demonstrate the importance of high-quality training data by achieving
higher grasping success rate over existing RL baselines using only tens of
demonstrations. WildLMa exploits CLIP for language-conditioned imitation
learning that empirically generalizes to objects unseen in training
demonstrations. Besides extensive quantitative evaluation, we qualitatively
demonstrate practical robot applications, such as cleaning up trash in
university hallways or outdoor terrains, operating articulated objects, and
rearranging items on a bookshelf.
WildLMa integrates learned low-level controllers, a skill library, and a planner to enable quadruped robots to perform complex manipulation tasks in diverse environments with minimal training data.
low-level controller VR-enabled whole-body teleoperation traversability visuomotor skills imitation learning heuristics learned skills LLM planners long-horizon tasks language-conditioned imitation learning CLIP grasping success rate RL baselines