ReVision: High-Quality, Low-Cost Video Generation with Explicit 3D
  Physics Modeling for Complex Motion and Interaction
In recent years, video generation has seen significant advancements. However,
challenges still persist in generating complex motions and interactions. To
address these challenges, we introduce ReVision, a plug-and-play framework that
explicitly integrates parameterized 3D physical knowledge into a pretrained
conditional video generation model, significantly enhancing its ability to
generate high-quality videos with complex motion and interactions.
Specifically, ReVision consists of three stages. First, a video diffusion model
is used to generate a coarse video. Next, we extract a set of 2D and 3D
features from the coarse video to construct a 3D object-centric representation,
which is then refined by our proposed parameterized physical prior model to
produce an accurate 3D motion sequence. Finally, this refined motion sequence
is fed back into the same video diffusion model as additional conditioning,
enabling the generation of motion-consistent videos, even in scenarios
involving complex actions and interactions. We validate the effectiveness of
our approach on Stable Video Diffusion, where ReVision significantly improves
motion fidelity and coherence. Remarkably, with only 1.5B parameters, it even
outperforms a state-of-the-art video generation model with over 13B parameters
on complex video generation by a substantial margin. Our results suggest that,
by incorporating 3D physical knowledge, even a relatively small video diffusion
model can generate complex motions and interactions with greater realism and
controllability, offering a promising solution for physically plausible video
generation.
6812db1567abbb1d11fb4e9d
Qihao Liu
admin_assigned
2025-05-05T08:14:06.173Z
639f1e519f1f2baab2f00d22
https://cdn-avatars.huggingface.co/v1/production/uploads/639f1e519f1f2baab2f00d22/pFjd51WZuVZ3A11rItvmk.jpeg
Qihao Liu
user
QHL067
6812db1567abbb1d11fb4e9e
Ju He
claimed_verified
2025-05-06T08:34:31.327Z
661c9059bcd78151e5c06ea1
https://cdn-avatars.huggingface.co/v1/production/uploads/661c9059bcd78151e5c06ea1/27bfNo1LZeZQ77vWuAa10.png
Ju He
user
turkeyju
6812db1567abbb1d11fb4e9f
Qihang Yu
6812db1567abbb1d11fb4ea0
Liang-Chieh Chen
6812db1567abbb1d11fb4ea1
Alan Yuille
ReVision: High-Quality, Low-Cost Video Generation with Explicit 3D
  Physics Modeling for Complex Motion and Interaction
In recent years, video generation has seen significant advancements. However,
challenges still persist in generating complex motions and interactions. To
address these challenges, we introduce ReVision, a plug-and-play framework that
explicitly integrates parameterized 3D physical knowledge into a pretrained
conditional video generation model, significantly enhancing its ability to
generate high-quality videos with complex motion and interactions.
Specifically, ReVision consists of three stages. First, a video diffusion model
is used to generate a coarse video. Next, we extract a set of 2D and 3D
features from the coarse video to construct a 3D object-centric representation,
which is then refined by our proposed parameterized physical prior model to
produce an accurate 3D motion sequence. Finally, this refined motion sequence
is fed back into the same video diffusion model as additional conditioning,
enabling the generation of motion-consistent videos, even in scenarios
involving complex actions and interactions. We validate the effectiveness of
our approach on Stable Video Diffusion, where ReVision significantly improves
motion fidelity and coherence. Remarkably, with only 1.5B parameters, it even
outperforms a state-of-the-art video generation model with over 13B parameters
on complex video generation by a substantial margin. Our results suggest that,
by incorporating 3D physical knowledge, even a relatively small video diffusion
model can generate complex motions and interactions with greater realism and
controllability, offering a promising solution for physically plausible video
generation.
ReVision enhances video generation by integrating 3D physical priors into video diffusion models, improving motion fidelity and coherence with fewer parameters.
video diffusion model 3D object-centric representation parameterized physical prior model motion-consistent videos motion fidelity coherence Stable Video Diffusion