STAR: Spatial-Temporal Augmentation with Text-to-Video Models for
  Real-World Video Super-Resolution
Image diffusion models have been adapted for real-world video
super-resolution to tackle over-smoothing issues in GAN-based methods. However,
these models struggle to maintain temporal consistency, as they are trained on
static images, limiting their ability to capture temporal dynamics effectively.
Integrating text-to-video (T2V) models into video super-resolution for improved
temporal modeling is straightforward. However, two key challenges remain:
artifacts introduced by complex degradations in real-world scenarios, and
compromised fidelity due to the strong generative capacity of powerful T2V
models (e.g., CogVideoX-5B). To enhance the spatio-temporal quality of
restored videos, we introduce~\name
(Spatial-Temporal Augmentation with T2V models for
Real-world video super-resolution), a novel approach that leverages
T2V models for real-world video super-resolution, achieving realistic spatial
details and robust temporal consistency. Specifically, we introduce a Local
Information Enhancement Module (LIEM) before the global attention block to
enrich local details and mitigate degradation artifacts. Moreover, we propose a
Dynamic Frequency (DF) Loss to reinforce fidelity, guiding the model to focus
on different frequency components across diffusion steps. Extensive experiments
demonstrate~\name~outperforms state-of-the-art methods on both
synthetic and real-world datasets.
677c986c1b9a7499c3644ef0
Rui Xie
claimed_verified
2025-01-07T08:41:59.722Z
660a7ecf14cfe4973e0acfe1
/avatars/e488058397f2b7a617515a4f721a9a00.svg
Rui Xie
user
SherryX
677c986c1b9a7499c3644ef1
Yinhong Liu
claimed_verified
2025-01-07T13:16:18.989Z
6548f92f5a65d02afb4a3021
/avatars/6bde44107ea0d1986356563f0738d2b8.svg
Yinhong Liu
user
Solitude-liu
677c986c1b9a7499c3644ef2
Penghao Zhou
claimed_verified
2025-01-07T13:02:04.376Z
667e5d480f66814a82e56420
/avatars/33fd797c7690959fed7449666a2502bf.svg
Zhou
user
LeonZhouph
677c986c1b9a7499c3644ef3
Chen Zhao
admin_assigned
2025-01-07T09:57:59.285Z
660103ec4ae78d4ded4633fc
/avatars/efce106d70f5d092bf44d0638aa49984.svg
CHEN Zhao
user
chenzhao
677c986c1b9a7499c3644ef4
Jun Zhou
677c986c1b9a7499c3644ef5
Kai Zhang
677c986c1b9a7499c3644ef6
Zhenyu Zhang
677c986c1b9a7499c3644ef7
Jian Yang
677c986c1b9a7499c3644ef8
Zhenheng Yang
admin_assigned
2025-01-07T09:58:14.804Z
6421183b69a2c2933882d652
/avatars/66813a8fa22915087cccd4dbfb945ca7.svg
Zhenheng Yang
user
zhenheny
677c986c1b9a7499c3644ef9
Ying Tai
admin_assigned
2025-01-07T09:58:21.644Z
65734004769f3ee9bde1af10
/avatars/d6310ed861972fd691687d8f47413f33.svg
Ying Tai
user
yingtai
STAR: Spatial-Temporal Augmentation with Text-to-Video Models for
  Real-World Video Super-Resolution
Image diffusion models have been adapted for real-world video
super-resolution to tackle over-smoothing issues in GAN-based methods. However,
these models struggle to maintain temporal consistency, as they are trained on
static images, limiting their ability to capture temporal dynamics effectively.
Integrating text-to-video (T2V) models into video super-resolution for improved
temporal modeling is straightforward. However, two key challenges remain:
artifacts introduced by complex degradations in real-world scenarios, and
compromised fidelity due to the strong generative capacity of powerful T2V
models (e.g., CogVideoX-5B). To enhance the spatio-temporal quality of
restored videos, we introduce~\name
(Spatial-Temporal Augmentation with T2V models for
Real-world video super-resolution), a novel approach that leverages
T2V models for real-world video super-resolution, achieving realistic spatial
details and robust temporal consistency. Specifically, we introduce a Local
Information Enhancement Module (LIEM) before the global attention block to
enrich local details and mitigate degradation artifacts. Moreover, we propose a
Dynamic Frequency (DF) Loss to reinforce fidelity, guiding the model to focus
on different frequency components across diffusion steps. Extensive experiments
demonstrate~\name~outperforms state-of-the-art methods on both
synthetic and real-world datasets.
A novel approach using text-to-video models with local information enhancement and dynamic frequency loss improves spatio-temporal quality in real-world video super-resolution.
image diffusion models T2V models video super-resolution temporal consistency Local Information Enhancement Module Dynamic Frequency Loss