PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual
  Observations
Modeling and rendering photorealistic avatars is of crucial importance in
many applications. Existing methods that build a 3D avatar from visual
observations, however, struggle to reconstruct clothed humans. We introduce
PhysAvatar, a novel framework that combines inverse rendering with inverse
physics to automatically estimate the shape and appearance of a human from
multi-view video data along with the physical parameters of the fabric of their
clothes. For this purpose, we adopt a mesh-aligned 4D Gaussian technique for
spatio-temporal mesh tracking as well as a physically based inverse renderer to
estimate the intrinsic material properties. PhysAvatar integrates a physics
simulator to estimate the physical parameters of the garments using
gradient-based optimization in a principled manner. These novel capabilities
enable PhysAvatar to create high-quality novel-view renderings of avatars
dressed in loose-fitting clothes under motions and lighting conditions not seen
in the training data. This marks a significant advancement towards modeling
photorealistic digital humans using physically based inverse rendering with
physics in the loop. Our project website is at:
https://qingqing-zhao.github.io/PhysAvatar
6614ca518cb1d029bac0f88d
Yang Zheng
6614ca518cb1d029bac0f88e
Qingqing Zhao
6614ca518cb1d029bac0f88f
Guandao Yang
claimed_verified
2024-04-25T06:55:08.036Z
64547ca006728ff79a38e7a5
https://cdn-avatars.huggingface.co/v1/production/uploads/64547ca006728ff79a38e7a5/xfTVQXMSfRv-IbljFpswQ.jpeg
Guandao Yang
user
guandao
6614ca518cb1d029bac0f890
Wang Yifan
6614ca518cb1d029bac0f891
Donglai Xiang
6614ca518cb1d029bac0f892
Florian Dubost
6614ca518cb1d029bac0f893
Dmitry Lagun
6614ca518cb1d029bac0f894
Thabo Beeler
6614ca518cb1d029bac0f895
Federico Tombari
6614ca518cb1d029bac0f896
Leonidas Guibas
6614ca518cb1d029bac0f897
Gordon Wetzstein
PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual
  Observations
Modeling and rendering photorealistic avatars is of crucial importance in
many applications. Existing methods that build a 3D avatar from visual
observations, however, struggle to reconstruct clothed humans. We introduce
PhysAvatar, a novel framework that combines inverse rendering with inverse
physics to automatically estimate the shape and appearance of a human from
multi-view video data along with the physical parameters of the fabric of their
clothes. For this purpose, we adopt a mesh-aligned 4D Gaussian technique for
spatio-temporal mesh tracking as well as a physically based inverse renderer to
estimate the intrinsic material properties. PhysAvatar integrates a physics
simulator to estimate the physical parameters of the garments using
gradient-based optimization in a principled manner. These novel capabilities
enable PhysAvatar to create high-quality novel-view renderings of avatars
dressed in loose-fitting clothes under motions and lighting conditions not seen
in the training data. This marks a significant advancement towards modeling
photorealistic digital humans using physically based inverse rendering with
physics in the loop. Our project website is at:
https://qingqing-zhao.github.io/PhysAvatar
PhysAvatar integrates inverse rendering and physics simulation to reconstruct and render realistic avatars in various dynamic and lighting conditions using multi-view video data.
inverse rendering inverse physics mesh-aligned 4D Gaussian spatio-temporal mesh tracking physically based inverse renderer physics simulator gradient-based optimization photorealistic digital humans