Open-World Skill Discovery from Unsegmented Demonstrations
Learning skills in open-world environments is essential for developing agents
capable of handling a variety of tasks by combining basic skills. Online
demonstration videos are typically long but unsegmented, making them difficult
to segment and label with skill identifiers. Unlike existing methods that rely
on sequence sampling or human labeling, we have developed a self-supervised
learning-based approach to segment these long videos into a series of
semantic-aware and skill-consistent segments. Drawing inspiration from human
cognitive event segmentation theory, we introduce Skill Boundary Detection
(SBD), an annotation-free temporal video segmentation algorithm. SBD detects
skill boundaries in a video by leveraging prediction errors from a pretrained
unconditional action-prediction model. This approach is based on the assumption
that a significant increase in prediction error indicates a shift in the skill
being executed. We evaluated our method in Minecraft, a rich open-world
simulator with extensive gameplay videos available online. Our SBD-generated
segments improved the average performance of conditioned policies by 63.7% and
52.1% on short-term atomic skill tasks, and their corresponding hierarchical
agents by 11.3% and 20.8% on long-horizon tasks. Our method can leverage the
diverse YouTube videos to train instruction-following agents. The project page
can be found in https://craftjarvis.github.io/SkillDiscovery.
67d8030568c586e2c8fcac8f
Jingwen Deng
claimed_verified
2025-03-17T14:42:10.438Z
66962a99a51ccecc90a662ae
/avatars/b4e200cc42eef5990678c168e6e51ab2.svg
DJW
user
fatty-belly
67d8030568c586e2c8fcac90
Zihao Wang
claimed_verified
2025-03-17T14:42:12.462Z
642e8c99c1b0f8e4e76bcaab
https://cdn-avatars.huggingface.co/v1/production/uploads/642e8c99c1b0f8e4e76bcaab/BOs9r0P9KyT9pEba9v0H4.png
Zihao Wang
user
zhwang4ai
67d8030568c586e2c8fcac91
Shaofei Cai
claimed_verified
2025-07-03T07:43:28.052Z
6578459d62d3ac1817ed79fe
https://cdn-avatars.huggingface.co/v1/production/uploads/6578459d62d3ac1817ed79fe/AXDJuwLUoEOb4Fj3U0Xxo.jpeg
Shaofei Cai
user
phython96
67d8030568c586e2c8fcac92
Anji Liu
67d8030568c586e2c8fcac93
Yitao Liang
claimed_verified
2025-04-09T07:38:16.563Z
64683a5776bb704aa14588b7
/avatars/e532756f52c5b95981470ace41a10556.svg
Yitao Liang
user
YitaoLiang
Open-World Skill Discovery from Unsegmented Demonstrations
Learning skills in open-world environments is essential for developing agents
capable of handling a variety of tasks by combining basic skills. Online
demonstration videos are typically long but unsegmented, making them difficult
to segment and label with skill identifiers. Unlike existing methods that rely
on sequence sampling or human labeling, we have developed a self-supervised
learning-based approach to segment these long videos into a series of
semantic-aware and skill-consistent segments. Drawing inspiration from human
cognitive event segmentation theory, we introduce Skill Boundary Detection
(SBD), an annotation-free temporal video segmentation algorithm. SBD detects
skill boundaries in a video by leveraging prediction errors from a pretrained
unconditional action-prediction model. This approach is based on the assumption
that a significant increase in prediction error indicates a shift in the skill
being executed. We evaluated our method in Minecraft, a rich open-world
simulator with extensive gameplay videos available online. Our SBD-generated
segments improved the average performance of conditioned policies by 63.7% and
52.1% on short-term atomic skill tasks, and their corresponding hierarchical
agents by 11.3% and 20.8% on long-horizon tasks. Our method can leverage the
diverse YouTube videos to train instruction-following agents. The project page
can be found in https://craftjarvis.github.io/SkillDiscovery.
A self-supervised Skill Boundary Detection method segments long, unlabelled demonstration videos into skill-consistent segments, enhancing performance in open-world simulations.
Skill Boundary Detection self-supervised learning temporal video segmentation prediction errors action-prediction model open-world environments Minecraft atomic skill tasks long-horizon tasks instruction-following agents