Video Diffusion Alignment via Reward Gradients
We have made significant progress towards building foundational video
diffusion models. As these models are trained using large-scale unsupervised
data, it has become crucial to adapt these models to specific downstream tasks.
Adapting these models via supervised fine-tuning requires collecting target
datasets of videos, which is challenging and tedious. In this work, we utilize
pre-trained reward models that are learned via preferences on top of powerful
vision discriminative models to adapt video diffusion models. These models
contain dense gradient information with respect to generated RGB pixels, which
is critical to efficient learning in complex search spaces, such as videos. We
show that backpropagating gradients from these reward models to a video
diffusion model can allow for compute and sample efficient alignment of the
video diffusion model. We show results across a variety of reward models and
video diffusion models, demonstrating that our approach can learn much more
efficiently in terms of reward queries and computation than prior gradient-free
approaches. Our code, model weights,and more visualization are available at
https://vader-vid.github.io.
6690952268b20cd8e62a7ca4
Mihir Prabhudesai
admin_assigned
2024-07-12T08:57:06.020Z
6310ff7dd43c55e811f8772f
/avatars/0dcab81771329d13cc62b08e55a71c9a.svg
Mihir Prabhudesai
user
mihirpd
6690952268b20cd8e62a7ca5
Russell Mendonca
6690952268b20cd8e62a7ca6
Zheyang Qin
admin_assigned
2024-07-12T09:25:47.447Z
65fe6f3a3d247974ba711d47
/avatars/c4f78f0cde2803458f4ab397e95ef652.svg
Zheyang Qin
user
zheyangqin
6690952268b20cd8e62a7ca7
Katerina Fragkiadaki
6690952268b20cd8e62a7ca8
Deepak Pathak
Video Diffusion Alignment via Reward Gradients
We have made significant progress towards building foundational video
diffusion models. As these models are trained using large-scale unsupervised
data, it has become crucial to adapt these models to specific downstream tasks.
Adapting these models via supervised fine-tuning requires collecting target
datasets of videos, which is challenging and tedious. In this work, we utilize
pre-trained reward models that are learned via preferences on top of powerful
vision discriminative models to adapt video diffusion models. These models
contain dense gradient information with respect to generated RGB pixels, which
is critical to efficient learning in complex search spaces, such as videos. We
show that backpropagating gradients from these reward models to a video
diffusion model can allow for compute and sample efficient alignment of the
video diffusion model. We show results across a variety of reward models and
video diffusion models, demonstrating that our approach can learn much more
efficiently in terms of reward queries and computation than prior gradient-free
approaches. Our code, model weights,and more visualization are available at
https://vader-vid.github.io.
Utilizing pre-trained reward models to adapt video diffusion models with gradient-based feedback enhances efficiency and performance compared to gradient-free methods.
video diffusion models pre-trained reward models vision discriminative models backpropagating gradients gradient information RGB pixels efficient learning reward queries computational efficiency