ReCamMaster: Camera-Controlled Generative Rendering from A Single Video
Camera control has been actively studied in text or image conditioned video
generation tasks. However, altering camera trajectories of a given video
remains under-explored, despite its importance in the field of video creation.
It is non-trivial due to the extra constraints of maintaining multiple-frame
appearance and dynamic synchronization. To address this, we present
ReCamMaster, a camera-controlled generative video re-rendering framework that
reproduces the dynamic scene of an input video at novel camera trajectories.
The core innovation lies in harnessing the generative capabilities of
pre-trained text-to-video models through a simple yet powerful video
conditioning mechanism -- its capability often overlooked in current research.
To overcome the scarcity of qualified training data, we construct a
comprehensive multi-camera synchronized video dataset using Unreal Engine 5,
which is carefully curated to follow real-world filming characteristics,
covering diverse scenes and camera movements. It helps the model generalize to
in-the-wild videos. Lastly, we further improve the robustness to diverse inputs
through a meticulously designed training strategy. Extensive experiments tell
that our method substantially outperforms existing state-of-the-art approaches
and strong baselines. Our method also finds promising applications in video
stabilization, super-resolution, and outpainting. Project page:
https://jianhongbai.github.io/ReCamMaster/
67d785fa473d4edd330edee1
Jianhong Bai
admin_assigned
2025-03-17T08:47:22.245Z
6530bf50f145530101ec03a2
/avatars/c61c00c314cf202b64968e51e855694d.svg
Jianhong Bai
user
jianhongbai
67d785fa473d4edd330edee2
Menghan Xia
admin_assigned
2025-03-17T08:47:41.792Z
63401c89f81b9d101361f712
https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
Richard
user
menghanxia
67d785fa473d4edd330edee3
Xiao Fu
67d785fa473d4edd330edee4
Xintao Wang
admin_assigned
2025-03-17T08:47:51.145Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
67d785fa473d4edd330edee5
Lianrui Mu
admin_assigned
2025-03-17T08:47:58.483Z
6672dd6d239ba86f129c5384
/avatars/6209afb551995b12d5e0d4d95e495694.svg
Lianrui Mu
user
Mu437
67d785fa473d4edd330edee6
Jinwen Cao
67d785fa473d4edd330edee7
Zuozhu Liu
admin_assigned
2025-03-17T08:48:17.243Z
6458b8d0990172cd1d703715
/avatars/55f0695e3cb9933c3903fde5a8f740d5.svg
Zuozhu Liu
user
Zuozhu
67d785fa473d4edd330edee8
Haoji Hu
admin_assigned
2025-03-17T08:48:23.846Z
66c46129d67297a9b93e03c5
/avatars/cffd8b07fa3655e240efc8e81f99d97d.svg
Haoji Hu
user
garland1979
67d785fa473d4edd330edee9
Xiang Bai
admin_assigned
2025-03-17T08:48:29.804Z
641790e2f1e86908935d82a0
/avatars/ced7a137c6344c74b7ac0d5c84833fc8.svg
Xiang Bai
user
baixianger
67d785fa473d4edd330edeea
Pengfei Wan
claimed_verified
2025-03-17T14:42:32.134Z
662f93942510ef5735d7ad00
https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
magicwpf
user
magicwpf
67d785fa473d4edd330edeeb
Di Zhang
admin_assigned
2025-03-17T08:48:49.559Z
644c8324f02250233d0d67d9
/avatars/feb39d281457c1750f3eada3c060a23e.svg
Di Zhang
user
dizhang
ReCamMaster: Camera-Controlled Generative Rendering from A Single Video
Camera control has been actively studied in text or image conditioned video
generation tasks. However, altering camera trajectories of a given video
remains under-explored, despite its importance in the field of video creation.
It is non-trivial due to the extra constraints of maintaining multiple-frame
appearance and dynamic synchronization. To address this, we present
ReCamMaster, a camera-controlled generative video re-rendering framework that
reproduces the dynamic scene of an input video at novel camera trajectories.
The core innovation lies in harnessing the generative capabilities of
pre-trained text-to-video models through a simple yet powerful video
conditioning mechanism -- its capability often overlooked in current research.
To overcome the scarcity of qualified training data, we construct a
comprehensive multi-camera synchronized video dataset using Unreal Engine 5,
which is carefully curated to follow real-world filming characteristics,
covering diverse scenes and camera movements. It helps the model generalize to
in-the-wild videos. Lastly, we further improve the robustness to diverse inputs
through a meticulously designed training strategy. Extensive experiments tell
that our method substantially outperforms existing state-of-the-art approaches
and strong baselines. Our method also finds promising applications in video
stabilization, super-resolution, and outpainting. Project page:
https://jianhongbai.github.io/ReCamMaster/
ReCamMaster uses pre-trained text-to-video models to render dynamic scenes of input videos from novel camera trajectories, leveraging a unique video conditioning mechanism and a custom dataset created with Unreal Engine 5.
camera-controlled generative video re-rendering pre-trained text-to-video models video conditioning mechanism Unreal Engine 5 video stabilization super-resolution outpainting