Reconstructing Animatable Categories from Videos
Building animatable 3D models is challenging due to the need for 3D scans,
laborious registration, and manual rigging, which are difficult to scale to
arbitrary categories. Recently, differentiable rendering provides a pathway to
obtain high-quality 3D models from monocular videos, but these are limited to
rigid categories or single instances. We present RAC that builds category 3D
models from monocular videos while disentangling variations over instances and
motion over time. Three key ideas are introduced to solve this problem: (1)
specializing a skeleton to instances via optimization, (2) a method for latent
space regularization that encourages shared structure across a category while
maintaining instance details, and (3) using 3D background models to disentangle
objects from the background. We show that 3D models of humans, cats, and dogs
can be learned from 50-100 internet videos.
645c64d8fbdc26a155fd4f6e
Gengshan Yang
admin_assigned
2023-05-24T09:03:40.546Z
62c0597543d4858f821bc87a
/avatars/ce9d28c08a5c9e8312a754714bae4634.svg
Gengshan Yang
user
gengshan-y
645c64d8fbdc26a155fd4f6f
Chaoyang Wang
645c64d8fbdc26a155fd4f70
N Dinesh Reddy
645c64d8fbdc26a155fd4f71
Deva Ramanan
admin_assigned
2023-05-24T09:03:49.674Z
6337151b0267ebcf02640eb6
/avatars/14a723cafc5587043bdfb19304fc202d.svg
Deva Ramanan
user
devakramanan
Reconstructing Animatable Categories from Videos
Building animatable 3D models is challenging due to the need for 3D scans,
laborious registration, and manual rigging, which are difficult to scale to
arbitrary categories. Recently, differentiable rendering provides a pathway to
obtain high-quality 3D models from monocular videos, but these are limited to
rigid categories or single instances. We present RAC that builds category 3D
models from monocular videos while disentangling variations over instances and
motion over time. Three key ideas are introduced to solve this problem: (1)
specializing a skeleton to instances via optimization, (2) a method for latent
space regularization that encourages shared structure across a category while
maintaining instance details, and (3) using 3D background models to disentangle
objects from the background. We show that 3D models of humans, cats, and dogs
can be learned from 50-100 internet videos.
RAC learns animatable 3D models from monocular videos by optimizing instance-specific skeletons, regularizing latent spaces, and disentangling objects from backgrounds, enabling category-wide models.
differentiable rendering monocular videos category 3D models variation disentangling motion disentangling skeleton optimization latent space regularization 3D background models