GARField: Group Anything with Radiance Fields
Grouping is inherently ambiguous due to the multiple levels of granularity in
which one can decompose a scene -- should the wheels of an excavator be
considered separate or part of the whole? We present Group Anything with
Radiance Fields (GARField), an approach for decomposing 3D scenes into a
hierarchy of semantically meaningful groups from posed image inputs. To do this
we embrace group ambiguity through physical scale: by optimizing a
scale-conditioned 3D affinity feature field, a point in the world can belong to
different groups of different sizes. We optimize this field from a set of 2D
masks provided by Segment Anything (SAM) in a way that respects coarse-to-fine
hierarchy, using scale to consistently fuse conflicting masks from different
viewpoints. From this field we can derive a hierarchy of possible groupings via
automatic tree construction or user interaction. We evaluate GARField on a
variety of in-the-wild scenes and find it effectively extracts groups at many
levels: clusters of objects, objects, and various subparts. GARField inherently
represents multi-view consistent groupings and produces higher fidelity groups
than the input SAM masks. GARField's hierarchical grouping could have exciting
downstream applications such as 3D asset extraction or dynamic scene
understanding. See the project website at https://www.garfield.studio/
65a8c02c9fa2a0f9a2b18e41
Chung Min Kim
admin_assigned
2024-01-18T10:14:17.239Z
6348a53762c668c7b48bd796
/avatars/a60214a0150b9c7047c6eaf482bc195b.svg
Chung Min Kim
user
chungmin99
65a8c02c9fa2a0f9a2b18e42
Mingxuan Wu
65a8c02c9fa2a0f9a2b18e43
Justin Kerr
admin_assigned
2024-01-18T10:14:32.002Z
63ec69436ef3ce22b88b636d
/avatars/83436c44a4650c8c81da769e024409a7.svg
Justin Kerr
user
wisepigeon
65a8c02c9fa2a0f9a2b18e44
Ken Goldberg
65a8c02c9fa2a0f9a2b18e45
Matthew Tancik
65a8c02c9fa2a0f9a2b18e46
Angjoo Kanazawa
admin_assigned
2024-01-18T10:14:49.447Z
6478cf7150ff7001631679c3
/avatars/65ec385a9cc44c972e6caf952e759ff1.svg
Angjoo Kanazawa
user
akanazawa
GARField: Group Anything with Radiance Fields
Grouping is inherently ambiguous due to the multiple levels of granularity in
which one can decompose a scene -- should the wheels of an excavator be
considered separate or part of the whole? We present Group Anything with
Radiance Fields (GARField), an approach for decomposing 3D scenes into a
hierarchy of semantically meaningful groups from posed image inputs. To do this
we embrace group ambiguity through physical scale: by optimizing a
scale-conditioned 3D affinity feature field, a point in the world can belong to
different groups of different sizes. We optimize this field from a set of 2D
masks provided by Segment Anything (SAM) in a way that respects coarse-to-fine
hierarchy, using scale to consistently fuse conflicting masks from different
viewpoints. From this field we can derive a hierarchy of possible groupings via
automatic tree construction or user interaction. We evaluate GARField on a
variety of in-the-wild scenes and find it effectively extracts groups at many
levels: clusters of objects, objects, and various subparts. GARField inherently
represents multi-view consistent groupings and produces higher fidelity groups
than the input SAM masks. GARField's hierarchical grouping could have exciting
downstream applications such as 3D asset extraction or dynamic scene
understanding. See the project website at https://www.garfield.studio/
GARField extracts semantically meaningful hierarchical groupings from 3D scenes using scale-conditioned 3D affinity feature fields derived from 2D masks.
scale-conditioned 3D affinity feature field Group Anything with Radiance Fields (GARField) Segment Anything (SAM) coarse-to-fine hierarchy automatic tree construction multi-view consistency 3D asset extraction dynamic scene understanding