Looking Backward: Streaming Video-to-Video Translation with Feature
  Banks
This paper introduces StreamV2V, a diffusion model that achieves real-time
streaming video-to-video (V2V) translation with user prompts. Unlike prior V2V
methods using batches to process limited frames, we opt to process frames in a
streaming fashion, to support unlimited frames. At the heart of StreamV2V lies
a backward-looking principle that relates the present to the past. This is
realized by maintaining a feature bank, which archives information from past
frames. For incoming frames, StreamV2V extends self-attention to include banked
keys and values and directly fuses similar past features into the output. The
feature bank is continually updated by merging stored and new features, making
it compact but informative. StreamV2V stands out for its adaptability and
efficiency, seamlessly integrating with image diffusion models without
fine-tuning. It can run 20 FPS on one A100 GPU, being 15x, 46x, 108x, and 158x
faster than FlowVid, CoDeF, Rerender, and TokenFlow, respectively. Quantitative
metrics and user studies confirm StreamV2V's exceptional ability to maintain
temporal consistency.
66555ecfc7ff1a0216abd811
Feng Liang
claimed_verified
2024-05-28T07:18:39.485Z
63631be58f43a912fc7595d5
https://cdn-avatars.huggingface.co/v1/production/uploads/1667440137801-63631be58f43a912fc7595d5.jpeg
Jeff Liang
user
JeffLiang
66555ecfc7ff1a0216abd812
Akio Kodaira
claimed_verified
2025-07-08T08:09:10.004Z
63fedca388b9695964c33ad8
https://cdn-avatars.huggingface.co/v1/production/uploads/63fedca388b9695964c33ad8/XFE8Tm9yRRbo6XdDw7c2O.png
Aki
user
AkiCumulo
66555ecfc7ff1a0216abd813
Chenfeng Xu
admin_assigned
2024-05-28T09:36:11.052Z
654ca71d5255ee86711b52c5
/avatars/52bf00fd74c8db5643c4daa185c678e6.svg
Chenfeng Xu
user
chenfengx
66555ecfc7ff1a0216abd814
Masayoshi Tomizuka
66555ecfc7ff1a0216abd815
Kurt Keutzer
admin_assigned
2024-05-28T09:36:25.609Z
6251bf4b183aa4266924ad91
https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
Kurt Keutzer
user
kurtkeutzer
66555ecfc7ff1a0216abd816
Diana Marculescu
Looking Backward: Streaming Video-to-Video Translation with Feature
  Banks
This paper introduces StreamV2V, a diffusion model that achieves real-time
streaming video-to-video (V2V) translation with user prompts. Unlike prior V2V
methods using batches to process limited frames, we opt to process frames in a
streaming fashion, to support unlimited frames. At the heart of StreamV2V lies
a backward-looking principle that relates the present to the past. This is
realized by maintaining a feature bank, which archives information from past
frames. For incoming frames, StreamV2V extends self-attention to include banked
keys and values and directly fuses similar past features into the output. The
feature bank is continually updated by merging stored and new features, making
it compact but informative. StreamV2V stands out for its adaptability and
efficiency, seamlessly integrating with image diffusion models without
fine-tuning. It can run 20 FPS on one A100 GPU, being 15x, 46x, 108x, and 158x
faster than FlowVid, CoDeF, Rerender, and TokenFlow, respectively. Quantitative
metrics and user studies confirm StreamV2V's exceptional ability to maintain
temporal consistency.
StreamV2V, a diffusion model for real-time video-to-video translation with user prompts, uses a streaming approach and a feature bank to fuse past frame information, achieving high frame rates and temporal consistency.
diffusion model streaming video-to-video (V2V) backward-looking principle feature bank self-attention temporal consistency