Fast Registration of Photorealistic Avatars for VR Facial Animation
Virtual Reality (VR) bares promise of social interactions that can feel more
immersive than other media. Key to this is the ability to accurately animate a
photorealistic avatar of one's likeness while wearing a VR headset. Although
high quality registration of person-specific avatars to headset-mounted camera
(HMC) images is possible in an offline setting, the performance of generic
realtime models are significantly degraded. Online registration is also
challenging due to oblique camera views and differences in modality. In this
work, we first show that the domain gap between the avatar and headset-camera
images is one of the primary sources of difficulty, where a transformer-based
architecture achieves high accuracy on domain-consistent data, but degrades
when the domain-gap is re-introduced. Building on this finding, we develop a
system design that decouples the problem into two parts: 1) an iterative
refinement module that takes in-domain inputs, and 2) a generic avatar-guided
image-to-image style transfer module that is conditioned on current estimation
of expression and head pose. These two modules reinforce each other, as image
style transfer becomes easier when close-to-ground-truth examples are shown,
and better domain-gap removal helps registration. Our system produces
high-quality results efficiently, obviating the need for costly offline
registration to generate personalized labels. We validate the accuracy and
efficiency of our approach through extensive experiments on a commodity
headset, demonstrating significant improvements over direct regression methods
as well as offline registration.
65af5993a134c07ddedaf223
Chaitanya Patel
claimed_verified
2024-10-13T20:25:16.873Z
66f10fa5f9cf1bf7e6798d15
/avatars/ff416c6778091eaf13f6c233a5e42244.svg
Chaitanya Patel
user
chaitanya100100
65af5993a134c07ddedaf224
Shaojie Bai
admin_assigned
2024-01-23T11:19:30.637Z
646a08304c1cd18b4975dfbe
/avatars/dfca388bb89deb82e874cd633c832a2c.svg
BaiShaojie
user
otnw
65af5993a134c07ddedaf225
Te-Li Wang
65af5993a134c07ddedaf226
Jason Saragih
65af5993a134c07ddedaf227
Shih-En Wei
Fast Registration of Photorealistic Avatars for VR Facial Animation
Virtual Reality (VR) bares promise of social interactions that can feel more
immersive than other media. Key to this is the ability to accurately animate a
photorealistic avatar of one's likeness while wearing a VR headset. Although
high quality registration of person-specific avatars to headset-mounted camera
(HMC) images is possible in an offline setting, the performance of generic
realtime models are significantly degraded. Online registration is also
challenging due to oblique camera views and differences in modality. In this
work, we first show that the domain gap between the avatar and headset-camera
images is one of the primary sources of difficulty, where a transformer-based
architecture achieves high accuracy on domain-consistent data, but degrades
when the domain-gap is re-introduced. Building on this finding, we develop a
system design that decouples the problem into two parts: 1) an iterative
refinement module that takes in-domain inputs, and 2) a generic avatar-guided
image-to-image style transfer module that is conditioned on current estimation
of expression and head pose. These two modules reinforce each other, as image
style transfer becomes easier when close-to-ground-truth examples are shown,
and better domain-gap removal helps registration. Our system produces
high-quality results efficiently, obviating the need for costly offline
registration to generate personalized labels. We validate the accuracy and
efficiency of our approach through extensive experiments on a commodity
headset, demonstrating significant improvements over direct regression methods
as well as offline registration.
A system for real-time avatar animation using VR cameras decouples domain gap removal and iterative refinement, improving accuracy and efficiency over existing methods.
transformer-based architecture image-to-image style transfer domain gap iterative refinement expression head pose