VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play
  Context Control
Video inpainting, which aims to restore corrupted video content, has
experienced substantial progress. Despite these advances, existing methods,
whether propagating unmasked region pixels through optical flow and receptive
field priors, or extending image-inpainting models temporally, face challenges
in generating fully masked objects or balancing the competing objectives of
background context preservation and foreground generation in one model,
respectively. To address these limitations, we propose a novel dual-stream
paradigm VideoPainter that incorporates an efficient context encoder
(comprising only 6% of the backbone parameters) to process masked videos and
inject backbone-aware background contextual cues to any pre-trained video DiT,
producing semantically consistent content in a plug-and-play manner. This
architectural separation significantly reduces the model's learning complexity
while enabling nuanced integration of crucial background context. We also
introduce a novel target region ID resampling technique that enables any-length
video inpainting, greatly enhancing our practical applicability. Additionally,
we establish a scalable dataset pipeline leveraging current vision
understanding models, contributing VPData and VPBench to facilitate
segmentation-based inpainting training and assessment, the largest video
inpainting dataset and benchmark to date with over 390K diverse clips. Using
inpainting as a pipeline basis, we also explore downstream applications
including video editing and video editing pair data generation, demonstrating
competitive performance and significant practical potential. Extensive
experiments demonstrate VideoPainter's superior performance in both any-length
video inpainting and editing, across eight key metrics, including video
quality, mask region preservation, and textual coherence.
67ce5ad85847e4787a82242d
Yuxuan Bian
extracted_pending
2025-03-10T03:22:04.947Z
650447dd52ca06fef957f05d
/avatars/511c11ac9b3cc7a162bda5e07f6ee0a3.svg
Yuxuan BIAN
user
BianYx
67ce5ad85847e4787a82242e
Zhaoyang Zhang
claimed_verified
2025-03-11T08:23:16.437Z
658409ceca19ccf6d9989add
/avatars/3ac1dbd25e52435185babdeb3da28875.svg
Zhaoyang Zhang
user
ZyZcuhk
67ce5ad85847e4787a82242f
Xuan Ju
admin_assigned
2025-03-10T09:57:06.955Z
62d4577bc85b0fcf7fde39bb
/avatars/a3a5729e33ae89ce9ba408830db3c835.svg
Xuan Ju
user
juxuan27
67ce5ad85847e4787a822430
Mingdeng Cao
admin_assigned
2025-03-10T09:56:45.412Z
6374a02d0856ac905bfc6113
/avatars/2cbe75c9cc818a647ca6e416f129c96f.svg
Mingdeng Cao
user
Ljzycmd
67ce5ad85847e4787a822431
Liangbin Xie
67ce5ad85847e4787a822432
Ying Shan
admin_assigned
2025-03-10T09:56:34.001Z
63ca3ddc04c979828310bfcb
/avatars/615e0d8622950b4408b40d550f02a894.svg
Ying Shan
user
yshan2u
67ce5ad85847e4787a822433
Qiang Xu
VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play
  Context Control
Video inpainting, which aims to restore corrupted video content, has
experienced substantial progress. Despite these advances, existing methods,
whether propagating unmasked region pixels through optical flow and receptive
field priors, or extending image-inpainting models temporally, face challenges
in generating fully masked objects or balancing the competing objectives of
background context preservation and foreground generation in one model,
respectively. To address these limitations, we propose a novel dual-stream
paradigm VideoPainter that incorporates an efficient context encoder
(comprising only 6% of the backbone parameters) to process masked videos and
inject backbone-aware background contextual cues to any pre-trained video DiT,
producing semantically consistent content in a plug-and-play manner. This
architectural separation significantly reduces the model's learning complexity
while enabling nuanced integration of crucial background context. We also
introduce a novel target region ID resampling technique that enables any-length
video inpainting, greatly enhancing our practical applicability. Additionally,
we establish a scalable dataset pipeline leveraging current vision
understanding models, contributing VPData and VPBench to facilitate
segmentation-based inpainting training and assessment, the largest video
inpainting dataset and benchmark to date with over 390K diverse clips. Using
inpainting as a pipeline basis, we also explore downstream applications
including video editing and video editing pair data generation, demonstrating
competitive performance and significant practical potential. Extensive
experiments demonstrate VideoPainter's superior performance in both any-length
video inpainting and editing, across eight key metrics, including video
quality, mask region preservation, and textual coherence.
VideoPainter uses a dual-stream paradigm with a context encoder to enhance video inpainting by balancing background context and foreground generation, supporting any-length inpainting and various downstream applications.
dual-stream paradigm context encoder video DiT backbone-aware semantically consistent content target region ID resampling segmentation-based inpainting VPData VPBench any-length video inpainting