Portrait Video Editing Empowered by Multimodal Generative Priors
We introduce PortraitGen, a powerful portrait video editing method that
achieves consistent and expressive stylization with multimodal prompts.
Traditional portrait video editing methods often struggle with 3D and temporal
consistency, and typically lack in rendering quality and efficiency. To address
these issues, we lift the portrait video frames to a unified dynamic 3D
Gaussian field, which ensures structural and temporal coherence across frames.
Furthermore, we design a novel Neural Gaussian Texture mechanism that not only
enables sophisticated style editing but also achieves rendering speed over
100FPS. Our approach incorporates multimodal inputs through knowledge distilled
from large-scale 2D generative models. Our system also incorporates expression
similarity guidance and a face-aware portrait editing module, effectively
mitigating degradation issues associated with iterative dataset updates.
Extensive experiments demonstrate the temporal consistency, editing efficiency,
and superior rendering quality of our method. The broad applicability of the
proposed approach is demonstrated through various applications, including
text-driven editing, image-driven editing, and relighting, highlighting its
great potential to advance the field of video editing. Demo videos and released
code are provided in our project page: https://ustc3dv.github.io/PortraitGen/
66f0c86916bffdccd54da55b
Xuan Gao
66f0c86916bffdccd54da55c
Haiyao Xiao
66f0c86916bffdccd54da55d
Chenglai Zhong
66f0c86916bffdccd54da55e
Shimin Hu
admin_assigned
2024-09-23T07:37:06.926Z
6522b0095e7247c2917269c0
/avatars/746db425094a0d997446e0c0fe53a733.svg
Hu Shimin
user
sisyphe28
66f0c86916bffdccd54da55f
Yudong Guo
claimed_verified
2024-09-24T07:53:18.422Z
6358a2cc1d66b442317f56c3
/avatars/119b991a9dafefce4b9217346044f7ea.svg
Yudong Guo
user
YudongGuo
66f0c86916bffdccd54da560
Juyong Zhang
Portrait Video Editing Empowered by Multimodal Generative Priors
We introduce PortraitGen, a powerful portrait video editing method that
achieves consistent and expressive stylization with multimodal prompts.
Traditional portrait video editing methods often struggle with 3D and temporal
consistency, and typically lack in rendering quality and efficiency. To address
these issues, we lift the portrait video frames to a unified dynamic 3D
Gaussian field, which ensures structural and temporal coherence across frames.
Furthermore, we design a novel Neural Gaussian Texture mechanism that not only
enables sophisticated style editing but also achieves rendering speed over
100FPS. Our approach incorporates multimodal inputs through knowledge distilled
from large-scale 2D generative models. Our system also incorporates expression
similarity guidance and a face-aware portrait editing module, effectively
mitigating degradation issues associated with iterative dataset updates.
Extensive experiments demonstrate the temporal consistency, editing efficiency,
and superior rendering quality of our method. The broad applicability of the
proposed approach is demonstrated through various applications, including
text-driven editing, image-driven editing, and relighting, highlighting its
great potential to advance the field of video editing. Demo videos and released
code are provided in our project page: https://ustc3dv.github.io/PortraitGen/
PortraitGen achieves consistent and expressive stylization in portrait video editing by using a 3D Gaussian field and a Neural Gaussian Texture mechanism, enabling sophisticated style editing and high rendering speed.
3D Gaussian field Neural Gaussian Texture temporal consistency multimodal inputs knowledge distillation expression similarity guidance face-aware portrait editing