VideoCanvas: Unified Video Completion from Arbitrary Spatiotemporal
  Patches via In-Context Conditioning
We introduce the task of arbitrary spatio-temporal video completion, where a
video is generated from arbitrary, user-specified patches placed at any spatial
location and timestamp, akin to painting on a video canvas. This flexible
formulation naturally unifies many existing controllable video generation
tasks--including first-frame image-to-video, inpainting, extension, and
interpolation--under a single, cohesive paradigm. Realizing this vision,
however, faces a fundamental obstacle in modern latent video diffusion models:
the temporal ambiguity introduced by causal VAEs, where multiple pixel frames
are compressed into a single latent representation, making precise frame-level
conditioning structurally difficult. We address this challenge with
VideoCanvas, a novel framework that adapts the In-Context Conditioning (ICC)
paradigm to this fine-grained control task with zero new parameters. We propose
a hybrid conditioning strategy that decouples spatial and temporal control:
spatial placement is handled via zero-padding, while temporal alignment is
achieved through Temporal RoPE Interpolation, which assigns each condition a
continuous fractional position within the latent sequence. This resolves the
VAE's temporal ambiguity and enables pixel-frame-aware control on a frozen
backbone. To evaluate this new capability, we develop VideoCanvasBench, the
first benchmark for arbitrary spatio-temporal video completion, covering both
intra-scene fidelity and inter-scene creativity. Experiments demonstrate that
VideoCanvas significantly outperforms existing conditioning paradigms,
establishing a new state of the art in flexible and unified video generation.
68e86e1995e8e6771df38919
Minghong Cai
claimed_verified
2025-10-10T04:32:50.095Z
64f94370c3c12b377cc51086
https://cdn-avatars.huggingface.co/v1/production/uploads/64f94370c3c12b377cc51086/6CXcHhqAoykqXcShqM8Rd.jpeg
Minghong Cai
user
onevfall
68e86e1995e8e6771df3891a
Qiulin Wang
68e86e1995e8e6771df3891b
Zongli Ye
68e86e1995e8e6771df3891c
Wenze Liu
68e86e1995e8e6771df3891d
Quande Liu
68e86e1995e8e6771df3891e
Weicai Ye
68e86e1995e8e6771df3891f
Xintao Wang
68e86e1995e8e6771df38920
Pengfei Wan
68e86e1995e8e6771df38921
Kun Gai
68e86e1995e8e6771df38922
Xiangyu Yue
VideoCanvas: Unified Video Completion from Arbitrary Spatiotemporal
  Patches via In-Context Conditioning
We introduce the task of arbitrary spatio-temporal video completion, where a
video is generated from arbitrary, user-specified patches placed at any spatial
location and timestamp, akin to painting on a video canvas. This flexible
formulation naturally unifies many existing controllable video generation
tasks--including first-frame image-to-video, inpainting, extension, and
interpolation--under a single, cohesive paradigm. Realizing this vision,
however, faces a fundamental obstacle in modern latent video diffusion models:
the temporal ambiguity introduced by causal VAEs, where multiple pixel frames
are compressed into a single latent representation, making precise frame-level
conditioning structurally difficult. We address this challenge with
VideoCanvas, a novel framework that adapts the In-Context Conditioning (ICC)
paradigm to this fine-grained control task with zero new parameters. We propose
a hybrid conditioning strategy that decouples spatial and temporal control:
spatial placement is handled via zero-padding, while temporal alignment is
achieved through Temporal RoPE Interpolation, which assigns each condition a
continuous fractional position within the latent sequence. This resolves the
VAE's temporal ambiguity and enables pixel-frame-aware control on a frozen
backbone. To evaluate this new capability, we develop VideoCanvasBench, the
first benchmark for arbitrary spatio-temporal video completion, covering both
intra-scene fidelity and inter-scene creativity. Experiments demonstrate that
VideoCanvas significantly outperforms existing conditioning paradigms,
establishing a new state of the art in flexible and unified video generation.
VideoCanvas addresses temporal ambiguity in latent video diffusion models to enable flexible spatio-temporal video completion using a hybrid conditioning strategy.
arbitrary spatio-temporal video completion latent video diffusion models causal VAEs VideoCanvas In-Context Conditioning (ICC) zero-padding Temporal RoPE Interpolation VideoCanvasBench intra-scene fidelity inter-scene creativity