Pixie: Fast and Generalizable Supervised Learning of 3D Physics from
  Pixels
Inferring the physical properties of 3D scenes from visual information is a
critical yet challenging task for creating interactive and realistic virtual
worlds. While humans intuitively grasp material characteristics such as
elasticity or stiffness, existing methods often rely on slow, per-scene
optimization, limiting their generalizability and application. To address this
problem, we introduce PIXIE, a novel method that trains a generalizable neural
network to predict physical properties across multiple scenes from 3D visual
features purely using supervised losses. Once trained, our feed-forward network
can perform fast inference of plausible material fields, which coupled with a
learned static scene representation like Gaussian Splatting enables realistic
physics simulation under external forces. To facilitate this research, we also
collected PIXIEVERSE, one of the largest known datasets of paired 3D assets and
physic material annotations. Extensive evaluations demonstrate that PIXIE is
about 1.46-4.39x better and orders of magnitude faster than test-time
optimization methods. By leveraging pretrained visual features like CLIP, our
method can also zero-shot generalize to real-world scenes despite only ever
been trained on synthetic data. https://pixie-3d.github.io/
68ae7587364411bea07df741
Long Le
claimed_verified
2025-08-27T07:10:46.864Z
664e94ba1b242206a6aa1583
/avatars/654b37699ba3f36f479274a2f372e7f8.svg
Long Le
user
vlongle
68ae7587364411bea07df742
Ryan Lucas
68ae7587364411bea07df743
Chen Wang
68ae7587364411bea07df744
Chuhao Chen
68ae7587364411bea07df745
Dinesh Jayaraman
68ae7587364411bea07df746
Eric Eaton
68ae7587364411bea07df747
Lingjie Liu
Pixie: Fast and Generalizable Supervised Learning of 3D Physics from
  Pixels
Inferring the physical properties of 3D scenes from visual information is a
critical yet challenging task for creating interactive and realistic virtual
worlds. While humans intuitively grasp material characteristics such as
elasticity or stiffness, existing methods often rely on slow, per-scene
optimization, limiting their generalizability and application. To address this
problem, we introduce PIXIE, a novel method that trains a generalizable neural
network to predict physical properties across multiple scenes from 3D visual
features purely using supervised losses. Once trained, our feed-forward network
can perform fast inference of plausible material fields, which coupled with a
learned static scene representation like Gaussian Splatting enables realistic
physics simulation under external forces. To facilitate this research, we also
collected PIXIEVERSE, one of the largest known datasets of paired 3D assets and
physic material annotations. Extensive evaluations demonstrate that PIXIE is
about 1.46-4.39x better and orders of magnitude faster than test-time
optimization methods. By leveraging pretrained visual features like CLIP, our
method can also zero-shot generalize to real-world scenes despite only ever
been trained on synthetic data. https://pixie-3d.github.io/
PIXIE, a neural network method, predicts physical properties of 3D scenes from visual features, enabling fast and realistic physics simulation using supervised learning and pretrained visual features.
neural network material fields Gaussian Splatting physics simulation PIXIEVERSE CLIP zero-shot generalization