One Shot, One Talk: Whole-body Talking Avatar from a Single Image
Building realistic and animatable avatars still requires minutes of
multi-view or monocular self-rotating videos, and most methods lack precise
control over gestures and expressions. To push this boundary, we address the
challenge of constructing a whole-body talking avatar from a single image. We
propose a novel pipeline that tackles two critical issues: 1) complex dynamic
modeling and 2) generalization to novel gestures and expressions. To achieve
seamless generalization, we leverage recent pose-guided image-to-video
diffusion models to generate imperfect video frames as pseudo-labels. To
overcome the dynamic modeling challenge posed by inconsistent and noisy
pseudo-videos, we introduce a tightly coupled 3DGS-mesh hybrid avatar
representation and apply several key regularizations to mitigate
inconsistencies caused by imperfect labels. Extensive experiments on diverse
subjects demonstrate that our method enables the creation of a photorealistic,
precisely animatable, and expressive whole-body talking avatar from just a
single image.
674e94bb320a043daeafc668
Jun Xiang
claimed_verified
2024-12-04T09:46:23.321Z
657d8495e50ca9a699ae7a1a
/avatars/afa2b9c30c37c1c2130cc3a2eabab19e.svg
Jun Xiang
user
xiangjun-xj
674e94bb320a043daeafc669
Yudong Guo
claimed_verified
2024-12-05T09:28:54.257Z
6358a2cc1d66b442317f56c3
/avatars/119b991a9dafefce4b9217346044f7ea.svg
Yudong Guo
user
YudongGuo
674e94bb320a043daeafc66a
Leipeng Hu
674e94bb320a043daeafc66b
Boyang Guo
674e94bb320a043daeafc66c
Yancheng Yuan
674e94bb320a043daeafc66d
Juyong Zhang
admin_assigned
2024-12-05T10:44:32.580Z
66fec5869b33512a6125ef2f
/avatars/5e9762ab35de5dd64ba06573fdea2ea7.svg
Juyong Zhang
user
JuyongZhang
One Shot, One Talk: Whole-body Talking Avatar from a Single Image
Building realistic and animatable avatars still requires minutes of
multi-view or monocular self-rotating videos, and most methods lack precise
control over gestures and expressions. To push this boundary, we address the
challenge of constructing a whole-body talking avatar from a single image. We
propose a novel pipeline that tackles two critical issues: 1) complex dynamic
modeling and 2) generalization to novel gestures and expressions. To achieve
seamless generalization, we leverage recent pose-guided image-to-video
diffusion models to generate imperfect video frames as pseudo-labels. To
overcome the dynamic modeling challenge posed by inconsistent and noisy
pseudo-videos, we introduce a tightly coupled 3DGS-mesh hybrid avatar
representation and apply several key regularizations to mitigate
inconsistencies caused by imperfect labels. Extensive experiments on diverse
subjects demonstrate that our method enables the creation of a photorealistic,
precisely animatable, and expressive whole-body talking avatar from just a
single image.
A new method creates photorealistic, expressive whole-body talking avatars from a single image using pose-guided image-to-video diffusion models and a hybrid 3DGS-mesh representation with regularizations.
pose-guided image-to-video diffusion models pseudo-labels 3DGS-mesh regularizations whole-body talking avatar