FürElise: Capturing and Physically Synthesizing Hand Motions of Piano
  Performance
Piano playing requires agile, precise, and coordinated hand control that
stretches the limits of dexterity. Hand motion models with the sophistication
to accurately recreate piano playing have a wide range of applications in
character animation, embodied AI, biomechanics, and VR/AR. In this paper, we
construct a first-of-its-kind large-scale dataset that contains approximately
10 hours of 3D hand motion and audio from 15 elite-level pianists playing 153
pieces of classical music. To capture natural performances, we designed a
markerless setup in which motions are reconstructed from multi-view videos
using state-of-the-art pose estimation models. The motion data is further
refined via inverse kinematics using the high-resolution MIDI key-pressing data
obtained from sensors in a specialized Yamaha Disklavier piano. Leveraging the
collected dataset, we developed a pipeline that can synthesize
physically-plausible hand motions for musical scores outside of the dataset.
Our approach employs a combination of imitation learning and reinforcement
learning to obtain policies for physics-based bimanual control involving the
interaction between hands and piano keys. To solve the sampling efficiency
problem with the large motion dataset, we use a diffusion model to generate
natural reference motions, which provide high-level trajectory and fingering
(finger order and placement) information. However, the generated reference
motion alone does not provide sufficient accuracy for piano performance
modeling. We then further augmented the data by using musical similarity to
retrieve similar motions from the captured dataset to boost the precision of
the RL policy. With the proposed method, our model generates natural, dexterous
motions that generalize to music from outside the training dataset.
670773181771817ab426144d
Ruocheng Wang
admin_assigned
2024-10-10T09:57:56.927Z
62abb35b26adcd4a3845b41b
/avatars/928720c54d947cc8ebcfa9a007f272ea.svg
Ruocheng Wang
user
rcwang
670773181771817ab426144e
Pei Xu
670773181771817ab426144f
Haochen Shi
670773181771817ab4261450
Elizabeth Schumann
670773181771817ab4261451
C. Karen Liu
FürElise: Capturing and Physically Synthesizing Hand Motions of Piano
  Performance
Piano playing requires agile, precise, and coordinated hand control that
stretches the limits of dexterity. Hand motion models with the sophistication
to accurately recreate piano playing have a wide range of applications in
character animation, embodied AI, biomechanics, and VR/AR. In this paper, we
construct a first-of-its-kind large-scale dataset that contains approximately
10 hours of 3D hand motion and audio from 15 elite-level pianists playing 153
pieces of classical music. To capture natural performances, we designed a
markerless setup in which motions are reconstructed from multi-view videos
using state-of-the-art pose estimation models. The motion data is further
refined via inverse kinematics using the high-resolution MIDI key-pressing data
obtained from sensors in a specialized Yamaha Disklavier piano. Leveraging the
collected dataset, we developed a pipeline that can synthesize
physically-plausible hand motions for musical scores outside of the dataset.
Our approach employs a combination of imitation learning and reinforcement
learning to obtain policies for physics-based bimanual control involving the
interaction between hands and piano keys. To solve the sampling efficiency
problem with the large motion dataset, we use a diffusion model to generate
natural reference motions, which provide high-level trajectory and fingering
(finger order and placement) information. However, the generated reference
motion alone does not provide sufficient accuracy for piano performance
modeling. We then further augmented the data by using musical similarity to
retrieve similar motions from the captured dataset to boost the precision of
the RL policy. With the proposed method, our model generates natural, dexterous
motions that generalize to music from outside the training dataset.
A system combining diffusion models and reinforcement learning generates realistic hand motions for piano playing from a large-scale dataset, enhancing generalization to new musical scores.
pose estimation models inverse kinematics MIDI key-pressing data diffusion model reinforcement learning physically-plausible hand motions bimanual control sampling efficiency musical similarity natural reference motions