GALA: Generating Animatable Layered Assets from a Single Scan
We present GALA, a framework that takes as input a single-layer clothed 3D
human mesh and decomposes it into complete multi-layered 3D assets. The outputs
can then be combined with other assets to create novel clothed human avatars
with any pose. Existing reconstruction approaches often treat clothed humans as
a single-layer of geometry and overlook the inherent compositionality of humans
with hairstyles, clothing, and accessories, thereby limiting the utility of the
meshes for downstream applications. Decomposing a single-layer mesh into
separate layers is a challenging task because it requires the synthesis of
plausible geometry and texture for the severely occluded regions. Moreover,
even with successful decomposition, meshes are not normalized in terms of poses
and body shapes, failing coherent composition with novel identities and poses.
To address these challenges, we propose to leverage the general knowledge of a
pretrained 2D diffusion model as geometry and appearance prior for humans and
other assets. We first separate the input mesh using the 3D surface
segmentation extracted from multi-view 2D segmentations. Then we synthesize the
missing geometry of different layers in both posed and canonical spaces using a
novel pose-guided Score Distillation Sampling (SDS) loss. Once we complete
inpainting high-fidelity 3D geometry, we also apply the same SDS loss to its
texture to obtain the complete appearance including the initially occluded
regions. Through a series of decomposition steps, we obtain multiple layers of
3D assets in a shared canonical space normalized in terms of poses and human
shapes, hence supporting effortless composition to novel identities and
reanimation with novel poses. Our experiments demonstrate the effectiveness of
our approach for decomposition, canonicalization, and composition tasks
compared to existing solutions.
65b08e9fa3085d018b4ba217
Taeksoo Kim
admin_assigned
2024-01-24T10:19:04.821Z
64b1673635c2e9909c266407
/avatars/2a0fdca5983dcaf1f2b09eac56cf586b.svg
Taeksoo Kim
user
Taeksoo
65b08e9fa3085d018b4ba218
Byungjun Kim
claimed_verified
2024-01-24T10:27:26.817Z
644f822623d7eb05ca691d95
/avatars/b9d3f39155ee96a79b8c873e4bab57e6.svg
Byungjun Kim
user
byungjun-kim
65b08e9fa3085d018b4ba219
Shunsuke Saito
admin_assigned
2024-01-24T10:19:37.621Z
630688edd37ce67e0e50e548
/avatars/1681e465d7649f67f94e1b69d236cb1e.svg
Shunsuke Saito
user
psyth
65b08e9fa3085d018b4ba21a
Hanbyul Joo
claimed_verified
2024-01-24T12:18:44.367Z
65b0ebe1165e69e0e6dc6ddd
/avatars/d2e95b2040c761200df1293db2949da7.svg
Hanbyul Joo
user
jhugestar
GALA: Generating Animatable Layered Assets from a Single Scan
We present GALA, a framework that takes as input a single-layer clothed 3D
human mesh and decomposes it into complete multi-layered 3D assets. The outputs
can then be combined with other assets to create novel clothed human avatars
with any pose. Existing reconstruction approaches often treat clothed humans as
a single-layer of geometry and overlook the inherent compositionality of humans
with hairstyles, clothing, and accessories, thereby limiting the utility of the
meshes for downstream applications. Decomposing a single-layer mesh into
separate layers is a challenging task because it requires the synthesis of
plausible geometry and texture for the severely occluded regions. Moreover,
even with successful decomposition, meshes are not normalized in terms of poses
and body shapes, failing coherent composition with novel identities and poses.
To address these challenges, we propose to leverage the general knowledge of a
pretrained 2D diffusion model as geometry and appearance prior for humans and
other assets. We first separate the input mesh using the 3D surface
segmentation extracted from multi-view 2D segmentations. Then we synthesize the
missing geometry of different layers in both posed and canonical spaces using a
novel pose-guided Score Distillation Sampling (SDS) loss. Once we complete
inpainting high-fidelity 3D geometry, we also apply the same SDS loss to its
texture to obtain the complete appearance including the initially occluded
regions. Through a series of decomposition steps, we obtain multiple layers of
3D assets in a shared canonical space normalized in terms of poses and human
shapes, hence supporting effortless composition to novel identities and
reanimation with novel poses. Our experiments demonstrate the effectiveness of
our approach for decomposition, canonicalization, and composition tasks
compared to existing solutions.
GALA uses a pretrained 2D diffusion model and pose-guided Score Distillation Sampling to decompose single-layer 3D human meshes into multi-layered assets for use in novel avatars.
GALA 2D diffusion model 3D surface segmentation multi-view 2D segmentations poses body shapes Score Distillation Sampling SDS loss high-fidelity 3D geometry canonical space decomposition canonicalization composition tasks