TextToon: Real-Time Text Toonify Head Avatar from Single Video
We propose TextToon, a method to generate a drivable toonified avatar. Given
a short monocular video sequence and a written instruction about the avatar
style, our model can generate a high-fidelity toonified avatar that can be
driven in real-time by another video with arbitrary identities. Existing
related works heavily rely on multi-view modeling to recover geometry via
texture embeddings, presented in a static manner, leading to control
limitations. The multi-view video input also makes it difficult to deploy these
models in real-world applications. To address these issues, we adopt a
conditional embedding Tri-plane to learn realistic and stylized facial
representations in a Gaussian deformation field. Additionally, we expand the
stylization capabilities of 3D Gaussian Splatting by introducing an adaptive
pixel-translation neural network and leveraging patch-aware contrastive
learning to achieve high-quality images. To push our work into consumer
applications, we develop a real-time system that can operate at 48 FPS on a GPU
machine and 15-18 FPS on a mobile machine. Extensive experiments demonstrate
the efficacy of our approach in generating textual avatars over existing
methods in terms of quality and real-time animation. Please refer to our
project page for more details: https://songluchuan.github.io/TextToon/.
67077a46be29abe37696f33a
Luchuan Song
claimed_verified
2024-10-10T14:03:10.468Z
6707dd1b5370cb53e70d673d
/avatars/795f443fa1f433a275e89e8ecd4cbfef.svg
Song
user
LuLuchuan
67077a46be29abe37696f33b
Lele Chen
67077a46be29abe37696f33c
Celong Liu
admin_assigned
2024-10-10T09:35:08.788Z
64364ba31adb261e94e0e5b4
/avatars/cb6ef2155d88b1e64c4ff36bd3a8f255.svg
Celong Liu
user
goddice
67077a46be29abe37696f33d
Pinxin Liu
admin_assigned
2024-10-10T09:35:03.031Z
62eb469dade76f18dd4f0dea
/avatars/c558254a0352d115a73febb90bb9370f.svg
Pinxin Liu
user
pliu23
67077a46be29abe37696f33e
Chenliang Xu
TextToon: Real-Time Text Toonify Head Avatar from Single Video
We propose TextToon, a method to generate a drivable toonified avatar. Given
a short monocular video sequence and a written instruction about the avatar
style, our model can generate a high-fidelity toonified avatar that can be
driven in real-time by another video with arbitrary identities. Existing
related works heavily rely on multi-view modeling to recover geometry via
texture embeddings, presented in a static manner, leading to control
limitations. The multi-view video input also makes it difficult to deploy these
models in real-world applications. To address these issues, we adopt a
conditional embedding Tri-plane to learn realistic and stylized facial
representations in a Gaussian deformation field. Additionally, we expand the
stylization capabilities of 3D Gaussian Splatting by introducing an adaptive
pixel-translation neural network and leveraging patch-aware contrastive
learning to achieve high-quality images. To push our work into consumer
applications, we develop a real-time system that can operate at 48 FPS on a GPU
machine and 15-18 FPS on a mobile machine. Extensive experiments demonstrate
the efficacy of our approach in generating textual avatars over existing
methods in terms of quality and real-time animation. Please refer to our
project page for more details: https://songluchuan.github.io/TextToon/.
TextToon generates high-fidelity, toonified avatars in real-time from a monocular video and text description using a conditional embedding Tri-plane, 3D Gaussian Splatting, and adaptive pixel-translation neural network.
monocular video toonified avatar real-time conditional embedding Tri-plane Gaussian deformation field 3D Gaussian Splatting adaptive pixel-translation neural network patch-aware contrastive learning real-time system 48 FPS 15-18 FPS