Pyramidal Flow Matching for Efficient Video Generative Modeling
Video generation requires modeling a vast spatiotemporal space, which demands
significant computational resources and data usage. To reduce the complexity,
the prevailing approaches employ a cascaded architecture to avoid direct
training with full resolution. Despite reducing computational demands, the
separate optimization of each sub-stage hinders knowledge sharing and
sacrifices flexibility. This work introduces a unified pyramidal flow matching
algorithm. It reinterprets the original denoising trajectory as a series of
pyramid stages, where only the final stage operates at the full resolution,
thereby enabling more efficient video generative modeling. Through our
sophisticated design, the flows of different pyramid stages can be interlinked
to maintain continuity. Moreover, we craft autoregressive video generation with
a temporal pyramid to compress the full-resolution history. The entire
framework can be optimized in an end-to-end manner and with a single unified
Diffusion Transformer (DiT). Extensive experiments demonstrate that our method
supports generating high-quality 5-second (up to 10-second) videos at 768p
resolution and 24 FPS within 20.7k A100 GPU training hours. All code and models
will be open-sourced at https://pyramid-flow.github.io.
67074f465ea108eecd170d48
Yang Jin
claimed_verified
2024-10-10T10:59:07.115Z
64b79ec0a8c39dc078897430
/avatars/8339f3ab9bb7baaf69bf174eafb7282c.svg
Yang Jin
user
rain1011
67074f465ea108eecd170d49
Zhicheng Sun
claimed_verified
2024-10-10T08:00:39.179Z
62fc758172a7ab50b4b89c8c
https://cdn-avatars.huggingface.co/v1/production/uploads/1677251053045-62fc758172a7ab50b4b89c8c.jpeg
Zhicheng Sun
user
feifeiobama
67074f465ea108eecd170d4a
Ningyuan Li
admin_assigned
2024-10-10T08:55:43.710Z
66af935fa20def3de33f54ed
/avatars/2666e97165377d979c530d96a1089d86.svg
Li Ningyuan
user
Ninggggy
67074f465ea108eecd170d4b
Kun Xu
67074f465ea108eecd170d4c
Kun Xu
67074f465ea108eecd170d4d
Hao Jiang
67074f465ea108eecd170d4e
Nan Zhuang
admin_assigned
2024-10-10T08:55:08.855Z
6461e1ab5e410d3ff8ec84e4
/avatars/16d5fc2994993ab65e8e43798cd864c4.svg
Nan Zhuang
user
Payne53
67074f465ea108eecd170d4f
Quzhe Huang
admin_assigned
2024-10-10T08:55:02.968Z
642fd69e8169f1dce23c0e64
/avatars/8dc14c7850b6274ab24906ea5b2e5e8a.svg
Quzhe Huang
user
quzhe
67074f465ea108eecd170d50
Yang Song
67074f465ea108eecd170d51
Yadong Mu
67074f465ea108eecd170d52
Zhouchen Lin
Pyramidal Flow Matching for Efficient Video Generative Modeling
Video generation requires modeling a vast spatiotemporal space, which demands
significant computational resources and data usage. To reduce the complexity,
the prevailing approaches employ a cascaded architecture to avoid direct
training with full resolution. Despite reducing computational demands, the
separate optimization of each sub-stage hinders knowledge sharing and
sacrifices flexibility. This work introduces a unified pyramidal flow matching
algorithm. It reinterprets the original denoising trajectory as a series of
pyramid stages, where only the final stage operates at the full resolution,
thereby enabling more efficient video generative modeling. Through our
sophisticated design, the flows of different pyramid stages can be interlinked
to maintain continuity. Moreover, we craft autoregressive video generation with
a temporal pyramid to compress the full-resolution history. The entire
framework can be optimized in an end-to-end manner and with a single unified
Diffusion Transformer (DiT). Extensive experiments demonstrate that our method
supports generating high-quality 5-second (up to 10-second) videos at 768p
resolution and 24 FPS within 20.7k A100 GPU training hours. All code and models
will be open-sourced at https://pyramid-flow.github.io.
A unified pyramidal flow matching algorithm with a single Diffusion Transformer enables efficient high-quality video generation by interlinking pyramid stages and compressing full-resolution history.
pyramidal flow matching denoising trajectory pyramid stages autoregressive video generation temporal pyramid diffusion transformer