PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable
  Objects from Videos
Creating a physical digital twin of a real-world object has immense potential
in robotics, content creation, and XR. In this paper, we present PhysTwin, a
novel framework that uses sparse videos of dynamic objects under interaction to
produce a photo- and physically realistic, real-time interactive virtual
replica. Our approach centers on two key components: (1) a physics-informed
representation that combines spring-mass models for realistic physical
simulation, generative shape models for geometry, and Gaussian splats for
rendering; and (2) a novel multi-stage, optimization-based inverse modeling
framework that reconstructs complete geometry, infers dense physical
properties, and replicates realistic appearance from videos. Our method
integrates an inverse physics framework with visual perception cues, enabling
high-fidelity reconstruction even from partial, occluded, and limited
viewpoints. PhysTwin supports modeling various deformable objects, including
ropes, stuffed animals, cloth, and delivery packages. Experiments show that
PhysTwin outperforms competing methods in reconstruction, rendering, future
prediction, and simulation under novel interactions. We further demonstrate its
applications in interactive real-time simulation and model-based robotic motion
planning.
67e35159b3f0452257728c31
Hanxiao Jiang
claimed_verified
2025-03-26T20:44:50.816Z
63f2bb987ddf724fbcc57ae9
https://cdn-avatars.huggingface.co/v1/production/uploads/63f2bb987ddf724fbcc57ae9/j81g8dBAMcJchjjsRQSrP.jpeg
Hanxiao Jiang
user
Jianghanxiao
67e35159b3f0452257728c32
Hao-Yu Hsu
67e35159b3f0452257728c33
Kaifeng Zhang
claimed_verified
2025-07-08T08:07:41.256Z
634eb9784677a5891c0bd967
/avatars/a2cdd27159bcc63aef0378fdfe8eb0f0.svg
Kaifeng Zhang
user
kaifz
67e35159b3f0452257728c34
Hsin-Ni Yu
67e35159b3f0452257728c35
Shenlong Wang
67e35159b3f0452257728c36
Yunzhu Li
PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable
  Objects from Videos
Creating a physical digital twin of a real-world object has immense potential
in robotics, content creation, and XR. In this paper, we present PhysTwin, a
novel framework that uses sparse videos of dynamic objects under interaction to
produce a photo- and physically realistic, real-time interactive virtual
replica. Our approach centers on two key components: (1) a physics-informed
representation that combines spring-mass models for realistic physical
simulation, generative shape models for geometry, and Gaussian splats for
rendering; and (2) a novel multi-stage, optimization-based inverse modeling
framework that reconstructs complete geometry, infers dense physical
properties, and replicates realistic appearance from videos. Our method
integrates an inverse physics framework with visual perception cues, enabling
high-fidelity reconstruction even from partial, occluded, and limited
viewpoints. PhysTwin supports modeling various deformable objects, including
ropes, stuffed animals, cloth, and delivery packages. Experiments show that
PhysTwin outperforms competing methods in reconstruction, rendering, future
prediction, and simulation under novel interactions. We further demonstrate its
applications in interactive real-time simulation and model-based robotic motion
planning.
PhysTwin is a framework using physics-informed and generative models to create a realistic digital twin from sparse video, supporting various deformable objects and real-time simulation.
physics-informed representation spring-mass models generative shape models Gaussian splats inverse modeling framework inverse physics framework visual perception cues deformable objects real-time simulation robotic motion planning