IMUSIC: IMU-based Facial Expression Capture
For facial motion capture and analysis, the dominated solutions are generally
based on visual cues, which cannot protect privacy and are vulnerable to
occlusions. Inertial measurement units (IMUs) serve as potential rescues yet
are mainly adopted for full-body motion capture. In this paper, we propose
IMUSIC to fill the gap, a novel path for facial expression capture using purely
IMU signals, significantly distant from previous visual solutions.The key
design in our IMUSIC is a trilogy. We first design micro-IMUs to suit facial
capture, companion with an anatomy-driven IMU placement scheme. Then, we
contribute a novel IMU-ARKit dataset, which provides rich paired IMU/visual
signals for diverse facial expressions and performances. Such unique
multi-modality brings huge potential for future directions like IMU-based
facial behavior analysis. Moreover, utilizing IMU-ARKit, we introduce a strong
baseline approach to accurately predict facial blendshape parameters from
purely IMU signals. Specifically, we tailor a Transformer diffusion model with
a two-stage training strategy for this novel tracking task. The IMUSIC
framework empowers us to perform accurate facial capture in scenarios where
visual methods falter and simultaneously safeguard user privacy. We conduct
extensive experiments about both the IMU configuration and technical components
to validate the effectiveness of our IMUSIC approach. Notably, IMUSIC enables
various potential and novel applications, i.e., privacy-protecting facial
capture, hybrid capture against occlusions, or detecting minute facial
movements that are often invisible through visual cues. We will release our
dataset and implementations to enrich more possibilities of facial capture and
analysis in our community.
65c307f3e272bf45aa1b45da
Youjia Wang
65c307f3e272bf45aa1b45db
Yiwen Wu
65c307f3e272bf45aa1b45dc
Ruiqian Li
65c307f3e272bf45aa1b45dd
Hengan Zhou
65c307f3e272bf45aa1b45de
Hongyang Lin
65c307f3e272bf45aa1b45df
Yingwenqi Jiang
65c307f3e272bf45aa1b45e0
Yingsheng Zhu
65c307f3e272bf45aa1b45e1
Guanpeng Long
65c307f3e272bf45aa1b45e2
Jingya Wang
65c307f3e272bf45aa1b45e3
Lan Xu
65c307f3e272bf45aa1b45e4
Jingyi Yu
IMUSIC: IMU-based Facial Expression Capture
For facial motion capture and analysis, the dominated solutions are generally
based on visual cues, which cannot protect privacy and are vulnerable to
occlusions. Inertial measurement units (IMUs) serve as potential rescues yet
are mainly adopted for full-body motion capture. In this paper, we propose
IMUSIC to fill the gap, a novel path for facial expression capture using purely
IMU signals, significantly distant from previous visual solutions.The key
design in our IMUSIC is a trilogy. We first design micro-IMUs to suit facial
capture, companion with an anatomy-driven IMU placement scheme. Then, we
contribute a novel IMU-ARKit dataset, which provides rich paired IMU/visual
signals for diverse facial expressions and performances. Such unique
multi-modality brings huge potential for future directions like IMU-based
facial behavior analysis. Moreover, utilizing IMU-ARKit, we introduce a strong
baseline approach to accurately predict facial blendshape parameters from
purely IMU signals. Specifically, we tailor a Transformer diffusion model with
a two-stage training strategy for this novel tracking task. The IMUSIC
framework empowers us to perform accurate facial capture in scenarios where
visual methods falter and simultaneously safeguard user privacy. We conduct
extensive experiments about both the IMU configuration and technical components
to validate the effectiveness of our IMUSIC approach. Notably, IMUSIC enables
various potential and novel applications, i.e., privacy-protecting facial
capture, hybrid capture against occlusions, or detecting minute facial
movements that are often invisible through visual cues. We will release our
dataset and implementations to enrich more possibilities of facial capture and
analysis in our community.
A novel IMU-based approach for facial expression capture, IMUSIC, uses a tailored micro-IMU architecture, a custom dataset, and a Transformer diffusion model to achieve high accuracy in privacy-protected scenarios and under occlusions.
IMU IMUSIC micro-IMUs anatomy-driven placement IMU-ARKit facial blendshape parameters Transformer diffusion model two-stage training strategy