SAMPart3D: Segment Any Part in 3D Objects
3D part segmentation is a crucial and challenging task in 3D perception,
playing a vital role in applications such as robotics, 3D generation, and 3D
editing. Recent methods harness the powerful Vision Language Models (VLMs) for
2D-to-3D knowledge distillation, achieving zero-shot 3D part segmentation.
However, these methods are limited by their reliance on text prompts, which
restricts the scalability to large-scale unlabeled datasets and the flexibility
in handling part ambiguities. In this work, we introduce SAMPart3D, a scalable
zero-shot 3D part segmentation framework that segments any 3D object into
semantic parts at multiple granularities, without requiring predefined part
label sets as text prompts. For scalability, we use text-agnostic vision
foundation models to distill a 3D feature extraction backbone, allowing scaling
to large unlabeled 3D datasets to learn rich 3D priors. For flexibility, we
distill scale-conditioned part-aware 3D features for 3D part segmentation at
multiple granularities. Once the segmented parts are obtained from the
scale-conditioned part-aware 3D features, we use VLMs to assign semantic labels
to each part based on the multi-view renderings. Compared to previous methods,
our SAMPart3D can scale to the recent large-scale 3D object dataset Objaverse
and handle complex, non-ordinary objects. Additionally, we contribute a new 3D
part segmentation benchmark to address the lack of diversity and complexity of
objects and parts in existing benchmarks. Experiments show that our SAMPart3D
significantly outperforms existing zero-shot 3D part segmentation methods, and
can facilitate various applications such as part-level editing and interactive
segmentation.
67341bc8bd4f7e2dfe198a26
Yunhan Yang
admin_assigned
2024-11-15T14:28:49.914Z
64b4eecf2fc8324fcb63b404
https://cdn-avatars.huggingface.co/v1/production/uploads/64b4eecf2fc8324fcb63b404/zGYqYVB4-o-GBMybJ8CDA.png
Yunhan Yang
user
yhyang-myron
67341bc8bd4f7e2dfe198a27
Yukun Huang
claimed_verified
2024-11-15T17:25:45.766Z
638ee900ee7e45e0474a5712
https://cdn-avatars.huggingface.co/v1/production/uploads/638ee900ee7e45e0474a5712/KLli_eCbWwffKR7oLDmV3.jpeg
Yukun Huang
user
KevinHuang
67341bc8bd4f7e2dfe198a28
Yuan-Chen Guo
admin_assigned
2024-11-15T14:32:02.999Z
6346aaa3f06b237ba4e297b0
/avatars/5acb986e993eab1461200f3e9d99d022.svg
Yuan-Chen Guo
user
bennyguo
67341bc8bd4f7e2dfe198a29
Liangjun Lu
admin_assigned
2024-11-15T14:32:13.410Z
664095e66246363bdd6acd7c
/avatars/f3d3a4d19f1f9397bcb9cb07afcafcfb.svg
luliangjun
user
kurtlu
67341bc8bd4f7e2dfe198a2a
Xiaoyang Wu
admin_assigned
2024-11-15T14:32:37.270Z
643e5d6a1d0e956d94bb3608
https://cdn-avatars.huggingface.co/v1/production/uploads/643e5d6a1d0e956d94bb3608/mTnu7Dts2RmDklHlp9Gqu.jpeg
Xiaoyang Wu
user
Gofinge
67341bc8bd4f7e2dfe198a2b
Edmund Y. Lam
67341bc8bd4f7e2dfe198a2c
Yan-Pei Cao
admin_assigned
2024-11-15T14:32:52.905Z
638066faf022c8a5803f7eb8
/avatars/4cfd699c3f6c5461b12b7dc5e3fe183d.svg
Yanpei Cao
user
pookiefoof
67341bc8bd4f7e2dfe198a2d
Xihui Liu
claimed_verified
2024-11-15T09:25:20.354Z
65d5ec74cd05bc1eaa125040
/avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
Xihui Liu
user
XihuiLiu
SAMPart3D: Segment Any Part in 3D Objects
3D part segmentation is a crucial and challenging task in 3D perception,
playing a vital role in applications such as robotics, 3D generation, and 3D
editing. Recent methods harness the powerful Vision Language Models (VLMs) for
2D-to-3D knowledge distillation, achieving zero-shot 3D part segmentation.
However, these methods are limited by their reliance on text prompts, which
restricts the scalability to large-scale unlabeled datasets and the flexibility
in handling part ambiguities. In this work, we introduce SAMPart3D, a scalable
zero-shot 3D part segmentation framework that segments any 3D object into
semantic parts at multiple granularities, without requiring predefined part
label sets as text prompts. For scalability, we use text-agnostic vision
foundation models to distill a 3D feature extraction backbone, allowing scaling
to large unlabeled 3D datasets to learn rich 3D priors. For flexibility, we
distill scale-conditioned part-aware 3D features for 3D part segmentation at
multiple granularities. Once the segmented parts are obtained from the
scale-conditioned part-aware 3D features, we use VLMs to assign semantic labels
to each part based on the multi-view renderings. Compared to previous methods,
our SAMPart3D can scale to the recent large-scale 3D object dataset Objaverse
and handle complex, non-ordinary objects. Additionally, we contribute a new 3D
part segmentation benchmark to address the lack of diversity and complexity of
objects and parts in existing benchmarks. Experiments show that our SAMPart3D
significantly outperforms existing zero-shot 3D part segmentation methods, and
can facilitate various applications such as part-level editing and interactive
segmentation.
SAMPart3D is a scalable and flexible zero-shot 3D part segmentation framework that leverages text-agnostic vision foundation models and multi-view renderings for semantic labeling.
Vision Language Models VLMs 3D feature extraction backbone scale-conditioned part-aware features multi-view renderings Objaverse 3D part segmentation benchmark part-level editing interactive segmentation