ChatAnything: Facetime Chat with LLM-Enhanced Personas
In this technical report, we target generating anthropomorphized personas for
LLM-based characters in an online manner, including visual appearance,
personality and tones, with only text descriptions. To achieve this, we first
leverage the in-context learning capability of LLMs for personality generation
by carefully designing a set of system prompts. We then propose two novel
concepts: the mixture of voices (MoV) and the mixture of diffusers (MoD) for
diverse voice and appearance generation. For MoV, we utilize the text-to-speech
(TTS) algorithms with a variety of pre-defined tones and select the most
matching one based on the user-provided text description automatically. For
MoD, we combine the recent popular text-to-image generation techniques and
talking head algorithms to streamline the process of generating talking
objects. We termed the whole framework as ChatAnything. With it, users could be
able to animate anything with any personas that are anthropomorphic using just
a few text inputs. However, we have observed that the anthropomorphic objects
produced by current generative models are often undetectable by pre-trained
face landmark detectors, leading to failure of the face motion generation, even
if these faces possess human-like appearances because those images are nearly
seen during the training (e.g., OOD samples). To address this issue, we
incorporate pixel-level guidance to infuse human face landmarks during the
image generation phase. To benchmark these metrics, we have built an evaluation
dataset. Based on it, we verify that the detection rate of the face landmark is
significantly increased from 57.0% to 92.5% thus allowing automatic face
animation based on generated speech content. The code and more results can be
found at https://chatanything.github.io/.
6552f3bb0166ff6bd22f4628
Yilin Zhao
admin_assigned
2023-11-14T10:04:59.786Z
6385d86181fe8c678a345ff0
https://cdn-avatars.huggingface.co/v1/production/uploads/6385d86181fe8c678a345ff0/ETvVOtBAQ-RpwfT4TCWsv.jpeg
Yilin Zhao
user
ermu2001
6552f3bb0166ff6bd22f4629
Xinbin Yuan
admin_assigned
2023-11-14T10:02:25.005Z
6512e44292a52e002ab78b6c
/avatars/16bc674177346d19e012c1f27dc8495a.svg
Yuanxinbin
user
xinbinLL
6552f3bb0166ff6bd22f462a
Shanghua Gao
6552f3bb0166ff6bd22f462b
Zhijie Lin
admin_assigned
2023-11-14T10:04:01.298Z
64415957bd0c9726529802f6
/avatars/1132d1ee68fb58ec635d57c8175caacd.svg
Zhijie Lin
user
Ikuinen
6552f3bb0166ff6bd22f462c
Qibin Hou
6552f3bb0166ff6bd22f462d
Jiashi Feng
6552f3bb0166ff6bd22f462e
Daquan Zhou
admin_assigned
2023-11-14T10:04:31.330Z
63fe0b160c1bbe8e29d2dd32
/avatars/bc574036287170a77057893efaa48e2d.svg
Zhou
user
DaQuan21
ChatAnything: Facetime Chat with LLM-Enhanced Personas
In this technical report, we target generating anthropomorphized personas for
LLM-based characters in an online manner, including visual appearance,
personality and tones, with only text descriptions. To achieve this, we first
leverage the in-context learning capability of LLMs for personality generation
by carefully designing a set of system prompts. We then propose two novel
concepts: the mixture of voices (MoV) and the mixture of diffusers (MoD) for
diverse voice and appearance generation. For MoV, we utilize the text-to-speech
(TTS) algorithms with a variety of pre-defined tones and select the most
matching one based on the user-provided text description automatically. For
MoD, we combine the recent popular text-to-image generation techniques and
talking head algorithms to streamline the process of generating talking
objects. We termed the whole framework as ChatAnything. With it, users could be
able to animate anything with any personas that are anthropomorphic using just
a few text inputs. However, we have observed that the anthropomorphic objects
produced by current generative models are often undetectable by pre-trained
face landmark detectors, leading to failure of the face motion generation, even
if these faces possess human-like appearances because those images are nearly
seen during the training (e.g., OOD samples). To address this issue, we
incorporate pixel-level guidance to infuse human face landmarks during the
image generation phase. To benchmark these metrics, we have built an evaluation
dataset. Based on it, we verify that the detection rate of the face landmark is
significantly increased from 57.0% to 92.5% thus allowing automatic face
animation based on generated speech content. The code and more results can be
found at https://chatanything.github.io/.
A framework for generating anthropomorphized personas with diverse voices and appearances from text descriptions using LLMs and generative models, with improved face landmark detection for automatic animation.
LLM-based characters in-context learning mixture of voices mixture of diffusers text-to-speech text-to-image talking head algorithms ChatAnything pixel-level guidance face landmark detection out-of-distribution (OOD) samples