Control4D: Dynamic Portrait Editing by Learning 4D GAN from 2D
  Diffusion-based Editor
Recent years have witnessed considerable achievements in editing images with
text instructions. When applying these editors to dynamic scene editing, the
new-style scene tends to be temporally inconsistent due to the frame-by-frame
nature of these 2D editors. To tackle this issue, we propose Control4D, a novel
approach for high-fidelity and temporally consistent 4D portrait editing.
Control4D is built upon an efficient 4D representation with a 2D
diffusion-based editor. Instead of using direct supervisions from the editor,
our method learns a 4D GAN from it and avoids the inconsistent supervision
signals. Specifically, we employ a discriminator to learn the generation
distribution based on the edited images and then update the generator with the
discrimination signals. For more stable training, multi-level information is
extracted from the edited images and used to facilitate the learning of the
generator. Experimental results show that Control4D surpasses previous
approaches and achieves more photo-realistic and consistent 4D editing
performances. The link to our project website is
https://control4darxiv.github.io.
6477f638e5ddd7d2a1739207
Ruizhi Shao
claimed_verified
2023-12-06T10:29:08.808Z
63f52856b51da4d61da7aa21
https://cdn-avatars.huggingface.co/v1/production/uploads/63f52856b51da4d61da7aa21/qSvSk1TnRp6DzdRzcoC8F.jpeg
Ruizhi Shao
user
Saurus
6477f638e5ddd7d2a1739208
Jingxiang Sun
6477f638e5ddd7d2a1739209
Cheng Peng
6477f638e5ddd7d2a173920a
Zerong Zheng
6477f638e5ddd7d2a173920b
Boyao Zhou
6477f638e5ddd7d2a173920c
Hongwen Zhang
admin_assigned
2023-06-02T12:29:51.421Z
638715a1aaee5cd69dc0c401
/avatars/f5976b3b02fe86191cb7f59c3dae102b.svg
Hongwen Zhang
user
HongwenZhang
6477f638e5ddd7d2a173920d
Yebin Liu
admin_assigned
2023-06-02T12:30:21.828Z
62e14dbe4db2175cd2735a80
/avatars/e6385dcedcb97e1b36281b49210321aa.svg
Yebin Liu
user
YebinLiu
Control4D: Dynamic Portrait Editing by Learning 4D GAN from 2D
  Diffusion-based Editor
Recent years have witnessed considerable achievements in editing images with
text instructions. When applying these editors to dynamic scene editing, the
new-style scene tends to be temporally inconsistent due to the frame-by-frame
nature of these 2D editors. To tackle this issue, we propose Control4D, a novel
approach for high-fidelity and temporally consistent 4D portrait editing.
Control4D is built upon an efficient 4D representation with a 2D
diffusion-based editor. Instead of using direct supervisions from the editor,
our method learns a 4D GAN from it and avoids the inconsistent supervision
signals. Specifically, we employ a discriminator to learn the generation
distribution based on the edited images and then update the generator with the
discrimination signals. For more stable training, multi-level information is
extracted from the edited images and used to facilitate the learning of the
generator. Experimental results show that Control4D surpasses previous
approaches and achieves more photo-realistic and consistent 4D editing
performances. The link to our project website is
https://control4darxiv.github.io.
Control4D uses a 2D diffusion-based editor within a 4D GAN framework to achieve high-fidelity and temporally consistent 4D portrait editing.
4D representation 2D diffusion-based editor 4D GAN discriminator generator multi-level information