LVCD: Reference-based Lineart Video Colorization with Diffusion Models
We propose the first video diffusion framework for reference-based lineart
video colorization. Unlike previous works that rely solely on image generative
models to colorize lineart frame by frame, our approach leverages a large-scale
pretrained video diffusion model to generate colorized animation videos. This
approach leads to more temporally consistent results and is better equipped to
handle large motions. Firstly, we introduce Sketch-guided ControlNet which
provides additional control to finetune an image-to-video diffusion model for
controllable video synthesis, enabling the generation of animation videos
conditioned on lineart. We then propose Reference Attention to facilitate the
transfer of colors from the reference frame to other frames containing fast and
expansive motions. Finally, we present a novel scheme for sequential sampling,
incorporating the Overlapped Blending Module and Prev-Reference Attention, to
extend the video diffusion model beyond its original fixed-length limitation
for long video colorization. Both qualitative and quantitative results
demonstrate that our method significantly outperforms state-of-the-art
techniques in terms of frame and video quality, as well as temporal
consistency. Moreover, our method is capable of generating high-quality, long
temporal-consistent animation videos with large motions, which is not
achievable in previous works. Our code and model are available at
https://luckyhzt.github.io/lvcd.
66ecded3381b190036da5ad8
Zhitong Huang
extracted_confirmed
2024-09-20T02:38:12.123Z
6321afb13007fcbf292ac1f4
/avatars/fb417553249580f1fb35bc127f550a5f.svg
Zhitong Huang
user
luckyhzt
66ecded3381b190036da5ad9
Mohan Zhang
extracted_pending
2024-09-26T01:24:55.221Z
66f227dd0c26057904ede944
/avatars/49ee01a747afd00ba8b337980d92cc64.svg
mohan zhang
user
zeromhzhang
66ecded3381b190036da5ada
Jing Liao
admin_assigned
2024-09-20T09:28:46.446Z
65e77726767bfc7d109c45bf
/avatars/24e68c86e06055ea1209598ba49ce8b9.svg
Jing Liao
user
CeciliaJL
LVCD: Reference-based Lineart Video Colorization with Diffusion Models
We propose the first video diffusion framework for reference-based lineart
video colorization. Unlike previous works that rely solely on image generative
models to colorize lineart frame by frame, our approach leverages a large-scale
pretrained video diffusion model to generate colorized animation videos. This
approach leads to more temporally consistent results and is better equipped to
handle large motions. Firstly, we introduce Sketch-guided ControlNet which
provides additional control to finetune an image-to-video diffusion model for
controllable video synthesis, enabling the generation of animation videos
conditioned on lineart. We then propose Reference Attention to facilitate the
transfer of colors from the reference frame to other frames containing fast and
expansive motions. Finally, we present a novel scheme for sequential sampling,
incorporating the Overlapped Blending Module and Prev-Reference Attention, to
extend the video diffusion model beyond its original fixed-length limitation
for long video colorization. Both qualitative and quantitative results
demonstrate that our method significantly outperforms state-of-the-art
techniques in terms of frame and video quality, as well as temporal
consistency. Moreover, our method is capable of generating high-quality, long
temporal-consistent animation videos with large motions, which is not
achievable in previous works. Our code and model are available at
https://luckyhzt.github.io/lvcd.
A video diffusion framework for lineart colorization uses Sketch-guided ControlNet, Reference Attention, and sequential sampling with Overlapped Blending Module and Prev-Reference Attention to produce temporally consistent, high-quality animation videos from lineart drawings.
video diffusion reference-based lineart image-to-video diffusion Sketch-guided ControlNet Reference Attention temporal consistency sequential sampling Overlapped Blending Module Prev-Reference Attention