GST: Precise 3D Human Body from a Single Image with Gaussian Splatting
  Transformers
Reconstructing realistic 3D human models from monocular images has
significant applications in creative industries, human-computer interfaces, and
healthcare. We base our work on 3D Gaussian Splatting (3DGS), a scene
representation composed of a mixture of Gaussians. Predicting such mixtures for
a human from a single input image is challenging, as it is a non-uniform
density (with a many-to-one relationship with input pixels) with strict
physical constraints. At the same time, it needs to be flexible to accommodate
a variety of clothes and poses. Our key observation is that the vertices of
standardized human meshes (such as SMPL) can provide an adequate density and
approximate initial position for Gaussians. We can then train a transformer
model to jointly predict comparatively small adjustments to these positions, as
well as the other Gaussians' attributes and the SMPL parameters. We show
empirically that this combination (using only multi-view supervision) can
achieve fast inference of 3D human models from a single image without test-time
optimization, expensive diffusion models, or 3D points supervision. We also
show that it can improve 3D pose estimation by better fitting human models that
account for clothes and other variations. The code is available on the project
website https://abdullahamdi.com/gst/ .
66decbb4ac5e6632f9067cba
Lorenza Prospero
claimed_verified
2024-09-10T08:43:47.840Z
66e0061d47e4afabbb0a35a0
/avatars/3c3fbcf4aacf6341f55ce8211c02df70.svg
Lorenza Prospero
user
prosperolo
66decbb4ac5e6632f9067cbb
Abdullah Hamdi
claimed_verified
2024-09-09T10:57:55.944Z
62fe3442e9061c0170d06e0b
https://cdn-avatars.huggingface.co/v1/production/uploads/1660827186084-62fe3442e9061c0170d06e0b.png
Abdullah Hamdi
user
ajhamdi
66decbb4ac5e6632f9067cbc
Joao F. Henriques
admin_assigned
2024-09-10T08:07:33.879Z
64bd4213891751ce9b17f4c3
/avatars/3a20ba60382a2b6fe5a576d4274ffc24.svg
Joao Henriques
user
JMHenry
66decbb4ac5e6632f9067cbd
Christian Rupprecht
GST: Precise 3D Human Body from a Single Image with Gaussian Splatting
  Transformers
Reconstructing realistic 3D human models from monocular images has
significant applications in creative industries, human-computer interfaces, and
healthcare. We base our work on 3D Gaussian Splatting (3DGS), a scene
representation composed of a mixture of Gaussians. Predicting such mixtures for
a human from a single input image is challenging, as it is a non-uniform
density (with a many-to-one relationship with input pixels) with strict
physical constraints. At the same time, it needs to be flexible to accommodate
a variety of clothes and poses. Our key observation is that the vertices of
standardized human meshes (such as SMPL) can provide an adequate density and
approximate initial position for Gaussians. We can then train a transformer
model to jointly predict comparatively small adjustments to these positions, as
well as the other Gaussians' attributes and the SMPL parameters. We show
empirically that this combination (using only multi-view supervision) can
achieve fast inference of 3D human models from a single image without test-time
optimization, expensive diffusion models, or 3D points supervision. We also
show that it can improve 3D pose estimation by better fitting human models that
account for clothes and other variations. The code is available on the project
website https://abdullahamdi.com/gst/ .
A transformer model predicts adjustments to Gaussian-splatted 3D human models from a single image, improving 3D pose estimation without test-time optimization or 3D point supervision.
3D Gaussian Splatting Gaussians SMPL transformer model multi-view supervision 3D human models 3D pose estimation