UniMMVSR: A Unified Multi-Modal Framework for Cascaded Video
  Super-Resolution
Cascaded video super-resolution has emerged as a promising technique for
decoupling the computational burden associated with generating high-resolution
videos using large foundation models. Existing studies, however, are largely
confined to text-to-video tasks and fail to leverage additional generative
conditions beyond text, which are crucial for ensuring fidelity in multi-modal
video generation. We address this limitation by presenting UniMMVSR, the first
unified generative video super-resolution framework to incorporate hybrid-modal
conditions, including text, images, and videos. We conduct a comprehensive
exploration of condition injection strategies, training schemes, and data
mixture techniques within a latent video diffusion model. A key challenge was
designing distinct data construction and condition utilization methods to
enable the model to precisely utilize all condition types, given their varied
correlations with the target video. Our experiments demonstrate that UniMMVSR
significantly outperforms existing methods, producing videos with superior
detail and a higher degree of conformity to multi-modal conditions. We also
validate the feasibility of combining UniMMVSR with a base model to achieve
multi-modal guided generation of 4K video, a feat previously unattainable with
existing techniques.
68e872cf95e8e6771df389dc
Shian Du
claimed_verified
2025-10-10T04:32:04.889Z
63fac5126b75d93aa136099c
/avatars/c1d7c6f29e67a6b2b97a7cd1cccedfb1.svg
Shian Du
user
YOKIMIYA
68e872cf95e8e6771df389dd
Menghan Xia
68e872cf95e8e6771df389de
Chang Liu
68e872cf95e8e6771df389df
Quande Liu
68e872cf95e8e6771df389e0
Xintao Wang
68e872cf95e8e6771df389e1
Pengfei Wan
68e872cf95e8e6771df389e2
Xiangyang Ji
UniMMVSR: A Unified Multi-Modal Framework for Cascaded Video
  Super-Resolution
Cascaded video super-resolution has emerged as a promising technique for
decoupling the computational burden associated with generating high-resolution
videos using large foundation models. Existing studies, however, are largely
confined to text-to-video tasks and fail to leverage additional generative
conditions beyond text, which are crucial for ensuring fidelity in multi-modal
video generation. We address this limitation by presenting UniMMVSR, the first
unified generative video super-resolution framework to incorporate hybrid-modal
conditions, including text, images, and videos. We conduct a comprehensive
exploration of condition injection strategies, training schemes, and data
mixture techniques within a latent video diffusion model. A key challenge was
designing distinct data construction and condition utilization methods to
enable the model to precisely utilize all condition types, given their varied
correlations with the target video. Our experiments demonstrate that UniMMVSR
significantly outperforms existing methods, producing videos with superior
detail and a higher degree of conformity to multi-modal conditions. We also
validate the feasibility of combining UniMMVSR with a base model to achieve
multi-modal guided generation of 4K video, a feat previously unattainable with
existing techniques.
UniMMVSR is a unified generative video super-resolution framework that incorporates hybrid-modal conditions, including text, images, and videos, within a latent video diffusion model, achieving superior detail and conformity to multi-modal conditions.
cascaded video super-resolution foundation models text-to-video tasks generative conditions multi-modal video generation unified generative video super-resolution hybrid-modal conditions latent video diffusion model condition injection strategies training schemes data mixture techniques multi-modal guided generation