VideoREPA: Learning Physics for Video Generation through Relational
  Alignment with Foundation Models
Recent advancements in text-to-video (T2V) diffusion models have enabled
high-fidelity and realistic video synthesis. However, current T2V models often
struggle to generate physically plausible content due to their limited inherent
ability to accurately understand physics. We found that while the
representations within T2V models possess some capacity for physics
understanding, they lag significantly behind those from recent video
self-supervised learning methods. To this end, we propose a novel framework
called VideoREPA, which distills physics understanding capability from video
understanding foundation models into T2V models by aligning token-level
relations. This closes the physics understanding gap and enable more
physics-plausible generation. Specifically, we introduce the Token Relation
Distillation (TRD) loss, leveraging spatio-temporal alignment to provide soft
guidance suitable for finetuning powerful pre-trained T2V models, a critical
departure from prior representation alignment (REPA) methods. To our knowledge,
VideoREPA is the first REPA method designed for finetuning T2V models and
specifically for injecting physical knowledge. Empirical evaluations show that
VideoREPA substantially enhances the physics commonsense of baseline method,
CogVideoX, achieving significant improvement on relevant benchmarks and
demonstrating a strong capacity for generating videos consistent with intuitive
physics. More video results are available at https://videorepa.github.io/.
6842520f05049fa51eed0e9f
Xiangdong Zhang
claimed_verified
2025-06-06T07:41:15.290Z
656d8d4b1f8d9b618de91369
/avatars/884dba9e56936241034b179d11a513b9.svg
Xiangdong Zhang
user
aHapBean
6842520f05049fa51eed0ea0
Jiaqi Liao
6842520f05049fa51eed0ea1
Shaofeng Zhang
6842520f05049fa51eed0ea2
Fanqing Meng
6842520f05049fa51eed0ea3
Xiangpeng Wan
6842520f05049fa51eed0ea4
Junchi Yan
6842520f05049fa51eed0ea5
Yu Cheng
VideoREPA: Learning Physics for Video Generation through Relational
  Alignment with Foundation Models
Recent advancements in text-to-video (T2V) diffusion models have enabled
high-fidelity and realistic video synthesis. However, current T2V models often
struggle to generate physically plausible content due to their limited inherent
ability to accurately understand physics. We found that while the
representations within T2V models possess some capacity for physics
understanding, they lag significantly behind those from recent video
self-supervised learning methods. To this end, we propose a novel framework
called VideoREPA, which distills physics understanding capability from video
understanding foundation models into T2V models by aligning token-level
relations. This closes the physics understanding gap and enable more
physics-plausible generation. Specifically, we introduce the Token Relation
Distillation (TRD) loss, leveraging spatio-temporal alignment to provide soft
guidance suitable for finetuning powerful pre-trained T2V models, a critical
departure from prior representation alignment (REPA) methods. To our knowledge,
VideoREPA is the first REPA method designed for finetuning T2V models and
specifically for injecting physical knowledge. Empirical evaluations show that
VideoREPA substantially enhances the physics commonsense of baseline method,
CogVideoX, achieving significant improvement on relevant benchmarks and
demonstrating a strong capacity for generating videos consistent with intuitive
physics. More video results are available at https://videorepa.github.io/.
VideoREPA enhances text-to-video synthesis by aligning token-level relations and distilling physics understanding from foundation models into T2V models.
T2V diffusion models physics understanding video self-supervised learning Token Relation Distillation (TRD) loss spatio-temporal alignment representation alignment (REPA) CogVideoX physics commonsense intuitive physics