Improving Video Generation with Human Feedback
Video generation has achieved significant advances through rectified flow
techniques, but issues like unsmooth motion and misalignment between videos and
prompts persist. In this work, we develop a systematic pipeline that harnesses
human feedback to mitigate these problems and refine the video generation
model. Specifically, we begin by constructing a large-scale human preference
dataset focused on modern video generation models, incorporating pairwise
annotations across multi-dimensions. We then introduce VideoReward, a
multi-dimensional video reward model, and examine how annotations and various
design choices impact its rewarding efficacy. From a unified reinforcement
learning perspective aimed at maximizing reward with KL regularization, we
introduce three alignment algorithms for flow-based models by extending those
from diffusion models. These include two training-time strategies: direct
preference optimization for flow (Flow-DPO) and reward weighted regression for
flow (Flow-RWR), and an inference-time technique, Flow-NRG, which applies
reward guidance directly to noisy videos. Experimental results indicate that
VideoReward significantly outperforms existing reward models, and Flow-DPO
demonstrates superior performance compared to both Flow-RWR and standard
supervised fine-tuning methods. Additionally, Flow-NRG lets users assign custom
weights to multiple objectives during inference, meeting personalized video
quality needs. Project page: https://gongyeliu.github.io/videoalign.
679319848d46289f90266168
Jie Liu
claimed_verified
2025-01-24T09:07:53.235Z
639be86b59473c6ae02ef9c4
https://cdn-avatars.huggingface.co/v1/production/uploads/639be86b59473c6ae02ef9c4/gw34RBCVZCOkcAA79xUr3.png
Jie Liu
user
jieliu
679319848d46289f90266169
Gongye Liu
claimed_verified
2025-02-05T13:37:43.546Z
6553316bf151de82f6a23e1d
https://cdn-avatars.huggingface.co/v1/production/uploads/6553316bf151de82f6a23e1d/GTBkSj4Fa3OoyM6Muz_Sc.jpeg
Gongye Liu
user
liuhuohuo
679319848d46289f9026616a
Jiajun Liang
679319848d46289f9026616b
Ziyang Yuan
679319848d46289f9026616c
Xiaokun Liu
679319848d46289f9026616d
Mingwu Zheng
679319848d46289f9026616e
Xiele Wu
679319848d46289f9026616f
Qiulin Wang
679319848d46289f90266170
Wenyu Qin
679319848d46289f90266171
Menghan Xia
679319848d46289f90266172
Xintao Wang
claimed_verified
2025-01-24T09:07:51.248Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
679319848d46289f90266173
Xiaohong Liu
679319848d46289f90266174
Fei Yang
679319848d46289f90266175
Pengfei Wan
claimed_verified
2025-03-17T14:42:51.336Z
662f93942510ef5735d7ad00
https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
magicwpf
user
magicwpf
679319848d46289f90266176
Di Zhang
679319848d46289f90266177
Kun Gai
679319848d46289f90266178
Yujiu Yang
679319848d46289f90266179
Wanli Ouyang
Improving Video Generation with Human Feedback
Video generation has achieved significant advances through rectified flow
techniques, but issues like unsmooth motion and misalignment between videos and
prompts persist. In this work, we develop a systematic pipeline that harnesses
human feedback to mitigate these problems and refine the video generation
model. Specifically, we begin by constructing a large-scale human preference
dataset focused on modern video generation models, incorporating pairwise
annotations across multi-dimensions. We then introduce VideoReward, a
multi-dimensional video reward model, and examine how annotations and various
design choices impact its rewarding efficacy. From a unified reinforcement
learning perspective aimed at maximizing reward with KL regularization, we
introduce three alignment algorithms for flow-based models by extending those
from diffusion models. These include two training-time strategies: direct
preference optimization for flow (Flow-DPO) and reward weighted regression for
flow (Flow-RWR), and an inference-time technique, Flow-NRG, which applies
reward guidance directly to noisy videos. Experimental results indicate that
VideoReward significantly outperforms existing reward models, and Flow-DPO
demonstrates superior performance compared to both Flow-RWR and standard
supervised fine-tuning methods. Additionally, Flow-NRG lets users assign custom
weights to multiple objectives during inference, meeting personalized video
quality needs. Project page: https://gongyeliu.github.io/videoalign.
VideoReward and alignment algorithms improve video generation by addressing unsmooth motion and misalignment using human feedback in a unified reinforcement learning framework.
rectified flow techniques human preference dataset VideoReward multi-dimensional video reward model reinforcement learning KL regularization flow-based models diffusion models direct preference optimization reward weighted regression reward guidance Flow-DPO Flow-RWR Flow-NRG standard supervised fine-tuning