ReCapture: Generative Video Camera Controls for User-Provided Videos
  using Masked Video Fine-Tuning
Recently, breakthroughs in video modeling have allowed for controllable
camera trajectories in generated videos. However, these methods cannot be
directly applied to user-provided videos that are not generated by a video
model. In this paper, we present ReCapture, a method for generating new videos
with novel camera trajectories from a single user-provided video. Our method
allows us to re-generate the reference video, with all its existing scene
motion, from vastly different angles and with cinematic camera motion. Notably,
using our method we can also plausibly hallucinate parts of the scene that were
not observable in the reference video. Our method works by (1) generating a
noisy anchor video with a new camera trajectory using multiview diffusion
models or depth-based point cloud rendering and then (2) regenerating the
anchor video into a clean and temporally consistent reangled video using our
proposed masked video fine-tuning technique.
672d74072dbda1cf55a7bd44
David Junhao Zhang
672d74072dbda1cf55a7bd45
Roni Paiss
672d74072dbda1cf55a7bd46
Shiran Zada
672d74072dbda1cf55a7bd47
Nikhil Karnad
672d74072dbda1cf55a7bd48
David E. Jacobs
672d74072dbda1cf55a7bd49
Yael Pritch
672d74072dbda1cf55a7bd4a
Inbar Mosseri
672d74072dbda1cf55a7bd4b
Mike Zheng Shou
672d74072dbda1cf55a7bd4c
Neal Wadhwa
672d74072dbda1cf55a7bd4d
Nataniel Ruiz
claimed_verified
2024-11-15T09:31:56.991Z
6318c6e6cb116eab31d70a16
https://cdn-avatars.huggingface.co/v1/production/uploads/6318c6e6cb116eab31d70a16/yAw5XtvqaPUkPj2B6JOzz.jpeg
Nataniel
user
flavoredquark
ReCapture: Generative Video Camera Controls for User-Provided Videos
  using Masked Video Fine-Tuning
Recently, breakthroughs in video modeling have allowed for controllable
camera trajectories in generated videos. However, these methods cannot be
directly applied to user-provided videos that are not generated by a video
model. In this paper, we present ReCapture, a method for generating new videos
with novel camera trajectories from a single user-provided video. Our method
allows us to re-generate the reference video, with all its existing scene
motion, from vastly different angles and with cinematic camera motion. Notably,
using our method we can also plausibly hallucinate parts of the scene that were
not observable in the reference video. Our method works by (1) generating a
noisy anchor video with a new camera trajectory using multiview diffusion
models or depth-based point cloud rendering and then (2) regenerating the
anchor video into a clean and temporally consistent reangled video using our
proposed masked video fine-tuning technique.
ReCapture generates new videos with different camera trajectories from a user-provided video using multiview diffusion models and masked video fine-tuning.
multiview diffusion models depth-based point cloud rendering masked video fine-tuning