Free^2Guide: Gradient-Free Path Integral Control for Enhancing
  Text-to-Video Generation with Large Vision-Language Models
Diffusion models have achieved impressive results in generative tasks like
text-to-image (T2I) and text-to-video (T2V) synthesis. However, achieving
accurate text alignment in T2V generation remains challenging due to the
complex temporal dependency across frames. Existing reinforcement learning
(RL)-based approaches to enhance text alignment often require differentiable
reward functions or are constrained to limited prompts, hindering their
scalability and applicability. In this paper, we propose Free^2Guide, a novel
gradient-free framework for aligning generated videos with text prompts without
requiring additional model training. Leveraging principles from path integral
control, Free^2Guide approximates guidance for diffusion models using
non-differentiable reward functions, thereby enabling the integration of
powerful black-box Large Vision-Language Models (LVLMs) as reward model.
Additionally, our framework supports the flexible ensembling of multiple reward
models, including large-scale image-based models, to synergistically enhance
alignment without incurring substantial computational overhead. We demonstrate
that Free^2Guide significantly improves text alignment across various
dimensions and enhances the overall quality of generated videos.
674952f5ad6772de5d595ae8
Jaemin Kim
claimed_verified
2024-11-29T09:33:20.413Z
64c3732de6c3860fba66ceb0
/avatars/785783ca08687923053eac641326281f.svg
JaeminKim
user
kjm981995
674952f5ad6772de5d595ae9
Bryan S Kim
claimed_verified
2025-05-29T20:18:21.966Z
6628efe14e1fa854f48d3a28
/avatars/aa5421149a07a82b5c2a25978f9b6926.svg
Sangwoo Kim
user
bryanswkim
674952f5ad6772de5d595aea
Jong Chul Ye
Free^2Guide: Gradient-Free Path Integral Control for Enhancing
  Text-to-Video Generation with Large Vision-Language Models
Diffusion models have achieved impressive results in generative tasks like
text-to-image (T2I) and text-to-video (T2V) synthesis. However, achieving
accurate text alignment in T2V generation remains challenging due to the
complex temporal dependency across frames. Existing reinforcement learning
(RL)-based approaches to enhance text alignment often require differentiable
reward functions or are constrained to limited prompts, hindering their
scalability and applicability. In this paper, we propose Free^2Guide, a novel
gradient-free framework for aligning generated videos with text prompts without
requiring additional model training. Leveraging principles from path integral
control, Free^2Guide approximates guidance for diffusion models using
non-differentiable reward functions, thereby enabling the integration of
powerful black-box Large Vision-Language Models (LVLMs) as reward model.
Additionally, our framework supports the flexible ensembling of multiple reward
models, including large-scale image-based models, to synergistically enhance
alignment without incurring substantial computational overhead. We demonstrate
that Free^2Guide significantly improves text alignment across various
dimensions and enhances the overall quality of generated videos.
Free$^2$Guide is a gradient-free framework that enhances text alignment in text-to-video generation using path integral control and non-differentiable reward functions, integrating large Vision-Language Models.
diffusion models text-to-image text-to-video reinforcement learning path integral control large Vision-Language Models reward models