Discovering and using Spelke segments
Segments in computer vision are often defined by semantic considerations and
are highly dependent on category-specific conventions. In contrast,
developmental psychology suggests that humans perceive the world in terms of
Spelke objects--groupings of physical things that reliably move together when
acted on by physical forces. Spelke objects thus operate on category-agnostic
causal motion relationships which potentially better support tasks like
manipulation and planning. In this paper, we first benchmark the Spelke object
concept, introducing the SpelkeBench dataset that contains a wide variety of
well-defined Spelke segments in natural images. Next, to extract Spelke
segments from images algorithmically, we build SpelkeNet, a class of visual
world models trained to predict distributions over future motions. SpelkeNet
supports estimation of two key concepts for Spelke object discovery: (1) the
motion affordance map, identifying regions likely to move under a poke, and (2)
the expected-displacement map, capturing how the rest of the scene will move.
These concepts are used for "statistical counterfactual probing", where diverse
"virtual pokes" are applied on regions of high motion-affordance, and the
resultant expected displacement maps are used define Spelke segments as
statistical aggregates of correlated motion statistics. We find that SpelkeNet
outperforms supervised baselines like SegmentAnything (SAM) on SpelkeBench.
Finally, we show that the Spelke concept is practically useful for downstream
applications, yielding superior performance on the 3DEditBench benchmark for
physical object manipulation when used in a variety of off-the-shelf object
manipulation models.
68833e9e846d2e78b7548bbf
Rahul Venkatesh
68833e9e846d2e78b7548bc0
Klemen Kotar
claimed_verified
2025-07-28T08:45:57.314Z
646d3cae4220471ca0c6cb13
/avatars/30e47db03a23d49d946cc5248b28959a.svg
Klemen Kotar
user
klemenk
68833e9e846d2e78b7548bc1
Lilian Naing Chen
68833e9e846d2e78b7548bc2
Seungwoo Kim
68833e9e846d2e78b7548bc3
Luca Thomas Wheeler
68833e9e846d2e78b7548bc4
Jared Watrous
68833e9e846d2e78b7548bc5
Ashley Xu
68833e9e846d2e78b7548bc6
Gia Ancone
68833e9e846d2e78b7548bc7
Wanhee Lee
68833e9e846d2e78b7548bc8
Honglin Chen
68833e9e846d2e78b7548bc9
Daniel Bear
68833e9e846d2e78b7548bca
Stefan Stojanov
68833e9e846d2e78b7548bcb
Daniel Yamins
Discovering and using Spelke segments
Segments in computer vision are often defined by semantic considerations and
are highly dependent on category-specific conventions. In contrast,
developmental psychology suggests that humans perceive the world in terms of
Spelke objects--groupings of physical things that reliably move together when
acted on by physical forces. Spelke objects thus operate on category-agnostic
causal motion relationships which potentially better support tasks like
manipulation and planning. In this paper, we first benchmark the Spelke object
concept, introducing the SpelkeBench dataset that contains a wide variety of
well-defined Spelke segments in natural images. Next, to extract Spelke
segments from images algorithmically, we build SpelkeNet, a class of visual
world models trained to predict distributions over future motions. SpelkeNet
supports estimation of two key concepts for Spelke object discovery: (1) the
motion affordance map, identifying regions likely to move under a poke, and (2)
the expected-displacement map, capturing how the rest of the scene will move.
These concepts are used for "statistical counterfactual probing", where diverse
"virtual pokes" are applied on regions of high motion-affordance, and the
resultant expected displacement maps are used define Spelke segments as
statistical aggregates of correlated motion statistics. We find that SpelkeNet
outperforms supervised baselines like SegmentAnything (SAM) on SpelkeBench.
Finally, we show that the Spelke concept is practically useful for downstream
applications, yielding superior performance on the 3DEditBench benchmark for
physical object manipulation when used in a variety of off-the-shelf object
manipulation models.
SpelkeNet, a visual world model, outperforms supervised baselines in identifying Spelke objects and enhances performance in physical object manipulation tasks.
Spelke objects SpelkeBench SpelkeNet visual world models motion affordance map expected-displacement map statistical counterfactual probing 3DEditBench