Cross-Frame Representation Alignment for Fine-Tuning Video Diffusion
  Models
Fine-tuning Video Diffusion Models (VDMs) at the user level to generate
videos that reflect specific attributes of training data presents notable
challenges, yet remains underexplored despite its practical importance.
Meanwhile, recent work such as Representation Alignment (REPA) has shown
promise in improving the convergence and quality of DiT-based image diffusion
models by aligning, or assimilating, its internal hidden states with external
pretrained visual features, suggesting its potential for VDM fine-tuning. In
this work, we first propose a straightforward adaptation of REPA for VDMs and
empirically show that, while effective for convergence, it is suboptimal in
preserving semantic consistency across frames. To address this limitation, we
introduce Cross-frame Representation Alignment (CREPA), a novel regularization
technique that aligns hidden states of a frame with external features from
neighboring frames. Empirical evaluations on large-scale VDMs, including
CogVideoX-5B and Hunyuan Video, demonstrate that CREPA improves both visual
fidelity and cross-frame semantic coherence when fine-tuned with
parameter-efficient methods such as LoRA. We further validate CREPA across
diverse datasets with varying attributes, confirming its broad applicability.
Project page: https://crepavideo.github.io
684ae250dbd21a9cc27b114c
Sungwon Hwang
684ae250dbd21a9cc27b114d
Hyojin Jang
684ae250dbd21a9cc27b114e
Kinam Kim
684ae250dbd21a9cc27b114f
Minho Park
684ae250dbd21a9cc27b1150
Jaegul choo
Cross-Frame Representation Alignment for Fine-Tuning Video Diffusion
  Models
Fine-tuning Video Diffusion Models (VDMs) at the user level to generate
videos that reflect specific attributes of training data presents notable
challenges, yet remains underexplored despite its practical importance.
Meanwhile, recent work such as Representation Alignment (REPA) has shown
promise in improving the convergence and quality of DiT-based image diffusion
models by aligning, or assimilating, its internal hidden states with external
pretrained visual features, suggesting its potential for VDM fine-tuning. In
this work, we first propose a straightforward adaptation of REPA for VDMs and
empirically show that, while effective for convergence, it is suboptimal in
preserving semantic consistency across frames. To address this limitation, we
introduce Cross-frame Representation Alignment (CREPA), a novel regularization
technique that aligns hidden states of a frame with external features from
neighboring frames. Empirical evaluations on large-scale VDMs, including
CogVideoX-5B and Hunyuan Video, demonstrate that CREPA improves both visual
fidelity and cross-frame semantic coherence when fine-tuned with
parameter-efficient methods such as LoRA. We further validate CREPA across
diverse datasets with varying attributes, confirming its broad applicability.
Project page: https://crepavideo.github.io
Cross-frame Representation Alignment (CREPA) enhances video diffusion model fine-tuning by improving visual fidelity and semantic coherence across frames using parameter-efficient methods.
fine-tuning Video Diffusion Models (VDMs) Representation Alignment (REPA) DiT-based image diffusion models Cross-frame Representation Alignment (CREPA) hidden states external features CogVideoX-5B Hunyuan Video visual fidelity cross-frame semantic coherence LoRA