URHand: Universal Relightable Hands
Existing photorealistic relightable hand models require extensive
identity-specific observations in different views, poses, and illuminations,
and face challenges in generalizing to natural illuminations and novel
identities. To bridge this gap, we present URHand, the first universal
relightable hand model that generalizes across viewpoints, poses,
illuminations, and identities. Our model allows few-shot personalization using
images captured with a mobile phone, and is ready to be photorealistically
rendered under novel illuminations. To simplify the personalization process
while retaining photorealism, we build a powerful universal relightable prior
based on neural relighting from multi-view images of hands captured in a light
stage with hundreds of identities. The key challenge is scaling the
cross-identity training while maintaining personalized fidelity and sharp
details without compromising generalization under natural illuminations. To
this end, we propose a spatially varying linear lighting model as the neural
renderer that takes physics-inspired shading as input feature. By removing
non-linear activations and bias, our specifically designed lighting model
explicitly keeps the linearity of light transport. This enables single-stage
training from light-stage data while generalizing to real-time rendering under
arbitrary continuous illuminations across diverse identities. In addition, we
introduce the joint learning of a physically based model and our neural
relighting model, which further improves fidelity and generalization. Extensive
experiments show that our approach achieves superior performance over existing
methods in terms of both quality and generalizability. We also demonstrate
quick personalization of URHand from a short phone scan of an unseen identity.
659f5cfd70cf8f1cbbf90e40
Zhaoxi Chen
admin_assigned
2024-01-11T10:39:37.408Z
62fc8cf7ee999004b5a8b982
/avatars/6c5dda9e58747054a989f077a078f3dc.svg
Zhaoxi Chen
user
FrozenBurning
659f5cfd70cf8f1cbbf90e41
Gyeongsik Moon
659f5cfd70cf8f1cbbf90e42
Kaiwen Guo
659f5cfd70cf8f1cbbf90e43
Chen Cao
659f5cfd70cf8f1cbbf90e44
Stanislav Pidhorskyi
659f5cfd70cf8f1cbbf90e45
Tomas Simon
659f5cfd70cf8f1cbbf90e46
Rohan Joshi
659f5cfd70cf8f1cbbf90e47
Yuan Dong
659f5cfd70cf8f1cbbf90e48
Yichen Xu
659f5cfd70cf8f1cbbf90e49
Bernardo Pires
659f5cfd70cf8f1cbbf90e4a
He Wen
659f5cfd70cf8f1cbbf90e4b
Lucas Evans
659f5cfd70cf8f1cbbf90e4c
Bo Peng
659f5cfd70cf8f1cbbf90e4d
Julia Buffalini
659f5cfd70cf8f1cbbf90e4e
Autumn Trimble
659f5cfd70cf8f1cbbf90e4f
Kevyn McPhail
659f5cfd70cf8f1cbbf90e50
Melissa Schoeller
659f5cfd70cf8f1cbbf90e51
Shoou-I Yu
659f5cfd70cf8f1cbbf90e52
Javier Romero
659f5cfd70cf8f1cbbf90e53
Michael Zollhöfer
659f5cfd70cf8f1cbbf90e54
Yaser Sheikh
659f5cfd70cf8f1cbbf90e55
Ziwei Liu
admin_assigned
2024-01-11T10:39:16.197Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
659f5cfd70cf8f1cbbf90e56
Shunsuke Saito
claimed_verified
2024-08-26T07:17:03.981Z
630688edd37ce67e0e50e548
/avatars/1681e465d7649f67f94e1b69d236cb1e.svg
Shunsuke Saito
user
psyth
URHand: Universal Relightable Hands
Existing photorealistic relightable hand models require extensive
identity-specific observations in different views, poses, and illuminations,
and face challenges in generalizing to natural illuminations and novel
identities. To bridge this gap, we present URHand, the first universal
relightable hand model that generalizes across viewpoints, poses,
illuminations, and identities. Our model allows few-shot personalization using
images captured with a mobile phone, and is ready to be photorealistically
rendered under novel illuminations. To simplify the personalization process
while retaining photorealism, we build a powerful universal relightable prior
based on neural relighting from multi-view images of hands captured in a light
stage with hundreds of identities. The key challenge is scaling the
cross-identity training while maintaining personalized fidelity and sharp
details without compromising generalization under natural illuminations. To
this end, we propose a spatially varying linear lighting model as the neural
renderer that takes physics-inspired shading as input feature. By removing
non-linear activations and bias, our specifically designed lighting model
explicitly keeps the linearity of light transport. This enables single-stage
training from light-stage data while generalizing to real-time rendering under
arbitrary continuous illuminations across diverse identities. In addition, we
introduce the joint learning of a physically based model and our neural
relighting model, which further improves fidelity and generalization. Extensive
experiments show that our approach achieves superior performance over existing
methods in terms of both quality and generalizability. We also demonstrate
quick personalization of URHand from a short phone scan of an unseen identity.
URHand is a universal relightable hand model that generalizes across different identities, viewpoints, and illuminations, using a spatially varying linear lighting model and joint learning of a physically based model for high-quality, real-time rendering.
neural relighting light stage spatially varying linear lighting model physics-inspired shading single-stage training physically based model joint learning