RDD: Retrieval-Based Demonstration Decomposer for Planner Alignment in Long-Horizon Tasks
Mingxuan Yan, Wang, Zechun Liu, Jiachen Li
To enable robots to achieve long-horizon tasks, recent hierarchical vision-language-action (VLAs) frameworks typically adopt vision-language model (VLM)-based planners to decompose complex manipulation tasks into simple sub-tasks that low-level visuomotor policies can easily handle. Typically, to finetune the VLM planner and let it learn to decompose the target task, a few human demonstrations are provided and will be segmented into sub-tasks by either human annotation or heuristic rules, which are less efficient, and the heuristic sub-tasks could largely deviate from the training data of visuomotor policy, which degrades the task performance. To address these issues, we propose a Retrieval-based Demonstration Decomposer (RDD) that automatically decomposes demonstrations into sub-tasks by aligning the visual features of decomposed sub-task intervals with the training data of low-level visuomotor policies to fully exploit its capability. Our method shows superior performance compared to the state-of-the-art sub-task decomposer on the RLBench benchmark and demonstrates robustness under various settings. Our code and demo videos are available in the supplementary materials.