GaussianProperty: Integrating Physical Properties to 3D Gaussians with
  LMMs
Estimating physical properties for visual data is a crucial task in computer
vision, graphics, and robotics, underpinning applications such as augmented
reality, physical simulation, and robotic grasping. However, this area remains
under-explored due to the inherent ambiguities in physical property estimation.
To address these challenges, we introduce GaussianProperty, a training-free
framework that assigns physical properties of materials to 3D Gaussians.
Specifically, we integrate the segmentation capability of SAM with the
recognition capability of GPT-4V(ision) to formulate a global-local physical
property reasoning module for 2D images. Then we project the physical
properties from multi-view 2D images to 3D Gaussians using a voting strategy.
We demonstrate that 3D Gaussians with physical property annotations enable
applications in physics-based dynamic simulation and robotic grasping. For
physics-based dynamic simulation, we leverage the Material Point Method (MPM)
for realistic dynamic simulation. For robot grasping, we develop a grasping
force prediction strategy that estimates a safe force range required for object
grasping based on the estimated physical properties. Extensive experiments on
material segmentation, physics-based dynamic simulation, and robotic grasping
validate the effectiveness of our proposed method, highlighting its crucial
role in understanding physical properties from visual data. Online demo, code,
more cases and annotated datasets are available on
https://Gaussian-Property.github.io{this https URL}.
6760e4c35b8363d75f3e7241
Xinli Xu
claimed_verified
2024-12-17T08:03:58.140Z
64b4ab62eec33e27dcd733b5
/avatars/0a9bf220c9a5efe7279f9b287b087d36.svg
Xinli XU
user
Xxlbigbrother
6760e4c35b8363d75f3e7242
Wenhang Ge
admin_assigned
2024-12-17T11:17:45.208Z
64758ee7d815855e4efa206b
/avatars/105ada08a9a982fc7b723bdc678f7e72.svg
wenhang ge
user
spongy
6760e4c35b8363d75f3e7243
Dicong Qiu
claimed_verified
2024-12-30T19:36:46.537Z
64c1f2bbc84cd12dd7a5468c
/avatars/d0a9fe33e5913b1304b8530b2c43f8bc.svg
David Qiu
user
davidqiu1993
6760e4c35b8363d75f3e7244
ZhiFei Chen
admin_assigned
2024-12-17T11:17:31.856Z
634a91c53a0cd2d4985dc98a
/avatars/fe680e683761c347cfa4bc9273b67b0d.svg
ZhiFei Chen
user
zhifeichen097
6760e4c35b8363d75f3e7245
Dongyu Yan
admin_assigned
2024-12-17T11:17:25.961Z
64049ae20ab5e22719f35103
https://cdn-avatars.huggingface.co/v1/production/uploads/1678023295407-noauth.jpeg
Dongyu Yan
user
StarYDY
6760e4c35b8363d75f3e7246
Zhuoyun Liu
6760e4c35b8363d75f3e7247
Haoyu Zhao
6760e4c35b8363d75f3e7248
Hanfeng Zhao
6760e4c35b8363d75f3e7249
Shunsi Zhang
6760e4c35b8363d75f3e724a
Junwei Liang
admin_assigned
2024-12-17T11:16:24.900Z
656eccea896aeb1168a2bade
/avatars/83298bc28c9a5de3d600b5d889c1dd82.svg
Junwei Liang
user
junweiliang
6760e4c35b8363d75f3e724b
Ying-Cong Chen
admin_assigned
2024-12-17T11:16:19.064Z
655cba1d87b67834000590e8
/avatars/3bd43b7c9351f65b8f38f4c8237a0146.svg
Yingcong Chen
user
yingcongchen
GaussianProperty: Integrating Physical Properties to 3D Gaussians with
  LMMs
Estimating physical properties for visual data is a crucial task in computer
vision, graphics, and robotics, underpinning applications such as augmented
reality, physical simulation, and robotic grasping. However, this area remains
under-explored due to the inherent ambiguities in physical property estimation.
To address these challenges, we introduce GaussianProperty, a training-free
framework that assigns physical properties of materials to 3D Gaussians.
Specifically, we integrate the segmentation capability of SAM with the
recognition capability of GPT-4V(ision) to formulate a global-local physical
property reasoning module for 2D images. Then we project the physical
properties from multi-view 2D images to 3D Gaussians using a voting strategy.
We demonstrate that 3D Gaussians with physical property annotations enable
applications in physics-based dynamic simulation and robotic grasping. For
physics-based dynamic simulation, we leverage the Material Point Method (MPM)
for realistic dynamic simulation. For robot grasping, we develop a grasping
force prediction strategy that estimates a safe force range required for object
grasping based on the estimated physical properties. Extensive experiments on
material segmentation, physics-based dynamic simulation, and robotic grasping
validate the effectiveness of our proposed method, highlighting its crucial
role in understanding physical properties from visual data. Online demo, code,
more cases and annotated datasets are available on
https://Gaussian-Property.github.io{this https URL}.
GaussianProperty is a training-free framework that assigns physical properties to 3D Gaussians using SAM and GPT-4V, enabling applications in physics-based dynamic simulation and robotic grasping.
GaussianProperty SAM GPT-4V 3D Gaussians physics-based dynamic simulation Material Point Method (MPM) robotic grasping grasping force prediction