Live2Diff: Live Stream Translation via Uni-directional Attention in
  Video Diffusion Models
Large Language Models have shown remarkable efficacy in generating streaming
data such as text and audio, thanks to their temporally uni-directional
attention mechanism, which models correlations between the current token and
previous tokens. However, video streaming remains much less explored, despite a
growing need for live video processing. State-of-the-art video diffusion models
leverage bi-directional temporal attention to model the correlations between
the current frame and all the surrounding (i.e. including future) frames, which
hinders them from processing streaming videos. To address this problem, we
present Live2Diff, the first attempt at designing a video diffusion model with
uni-directional temporal attention, specifically targeting live streaming video
translation. Compared to previous works, our approach ensures temporal
consistency and smoothness by correlating the current frame with its
predecessors and a few initial warmup frames, without any future frames.
Additionally, we use a highly efficient denoising scheme featuring a KV-cache
mechanism and pipelining, to facilitate streaming video translation at
interactive framerates. Extensive experiments demonstrate the effectiveness of
the proposed attention mechanism and pipeline, outperforming previous methods
in terms of temporal smoothness and/or efficiency.
66909203b70d356ed3bf03a7
Zhening Xing
claimed_verified
2024-07-12T07:40:11.783Z
62fb2a9dc95d426ff8f74c8d
/avatars/25c1a68ee7b7d0cc7e9f56bde37f4914.svg
Zhening Xing
user
Leoxing
66909203b70d356ed3bf03a8
Gereon Fox
admin_assigned
2024-07-12T12:36:03.149Z
6615447fa15c52fa7aa5d4b3
/avatars/fbdefe0a848ce65528dd38829e111072.svg
Gereon Fox
user
gfhcs
66909203b70d356ed3bf03a9
Yanhong Zeng
admin_assigned
2024-07-12T12:36:10.050Z
63d4b843df01ef426a0f79fb
https://cdn-avatars.huggingface.co/v1/production/uploads/1676365795587-63d4b843df01ef426a0f79fb.jpeg
Yanhong Zeng
user
zengyh1900
66909203b70d356ed3bf03aa
Xingang Pan
admin_assigned
2024-07-12T12:37:14.896Z
646758d6a9b4610868a138ff
https://cdn-avatars.huggingface.co/v1/production/uploads/646758d6a9b4610868a138ff/fNBlK5g4g7vlPAR1bQimZ.png
Xingang Pan
user
Xingang-Pan
66909203b70d356ed3bf03ab
Mohamed Elgharib
66909203b70d356ed3bf03ac
Christian Theobalt
66909203b70d356ed3bf03ad
Kai Chen
Live2Diff: Live Stream Translation via Uni-directional Attention in
  Video Diffusion Models
Large Language Models have shown remarkable efficacy in generating streaming
data such as text and audio, thanks to their temporally uni-directional
attention mechanism, which models correlations between the current token and
previous tokens. However, video streaming remains much less explored, despite a
growing need for live video processing. State-of-the-art video diffusion models
leverage bi-directional temporal attention to model the correlations between
the current frame and all the surrounding (i.e. including future) frames, which
hinders them from processing streaming videos. To address this problem, we
present Live2Diff, the first attempt at designing a video diffusion model with
uni-directional temporal attention, specifically targeting live streaming video
translation. Compared to previous works, our approach ensures temporal
consistency and smoothness by correlating the current frame with its
predecessors and a few initial warmup frames, without any future frames.
Additionally, we use a highly efficient denoising scheme featuring a KV-cache
mechanism and pipelining, to facilitate streaming video translation at
interactive framerates. Extensive experiments demonstrate the effectiveness of
the proposed attention mechanism and pipeline, outperforming previous methods
in terms of temporal smoothness and/or efficiency.
Live2Diff is a video diffusion model with uni-directional temporal attention, enabling efficient live streaming video translation with improved temporal smoothness.
video diffusion models bi-directional temporal attention uni-directional temporal attention live streaming video translation KV-cache mechanism pipelining