Hierarchical Patch Diffusion Models for High-Resolution Video Generation
Diffusion models have demonstrated remarkable performance in image and video
synthesis. However, scaling them to high-resolution inputs is challenging and
requires restructuring the diffusion pipeline into multiple independent
components, limiting scalability and complicating downstream applications. This
makes it very efficient during training and unlocks end-to-end optimization on
high-resolution videos. We improve PDMs in two principled ways. First, to
enforce consistency between patches, we develop deep context fusion -- an
architectural technique that propagates the context information from low-scale
to high-scale patches in a hierarchical manner. Second, to accelerate training
and inference, we propose adaptive computation, which allocates more network
capacity and computation towards coarse image details. The resulting model sets
a new state-of-the-art FVD score of 66.32 and Inception Score of 87.68 in
class-conditional video generation on UCF-101 256^2, surpassing recent
methods by more than 100%. Then, we show that it can be rapidly fine-tuned from
a base 36times 64 low-resolution generator for high-resolution 64 times
288 times 512 text-to-video synthesis. To the best of our knowledge, our
model is the first diffusion-based architecture which is trained on such high
resolutions entirely end-to-end. Project webpage:
https://snap-research.github.io/hpdm.
666a53d7372850ab364c4034
Ivan Skorokhodov
admin_assigned
2024-06-13T20:17:42.648Z
63610db13c7147fae7de88e3
/avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
Ivan Skorokhodov
user
universome
666a53d7372850ab364c4035
Willi Menapace
admin_assigned
2024-06-13T20:17:48.025Z
6315358a362e3e95ea538081
/avatars/3b089a25a87c2e83c6b23ccb5d2dc73e.svg
Willi Menapace
user
willi-menapace
666a53d7372850ab364c4036
Aliaksandr Siarohin
admin_assigned
2024-06-13T20:17:54.235Z
64276311eb9a0ed86180715b
/avatars/76f933cd549f10e5e2db379de235d304.svg
Aliaksandr Siarohin
user
aliaksandr-siarohin
666a53d7372850ab364c4037
Sergey Tulyakov
Hierarchical Patch Diffusion Models for High-Resolution Video Generation
Diffusion models have demonstrated remarkable performance in image and video
synthesis. However, scaling them to high-resolution inputs is challenging and
requires restructuring the diffusion pipeline into multiple independent
components, limiting scalability and complicating downstream applications. This
makes it very efficient during training and unlocks end-to-end optimization on
high-resolution videos. We improve PDMs in two principled ways. First, to
enforce consistency between patches, we develop deep context fusion -- an
architectural technique that propagates the context information from low-scale
to high-scale patches in a hierarchical manner. Second, to accelerate training
and inference, we propose adaptive computation, which allocates more network
capacity and computation towards coarse image details. The resulting model sets
a new state-of-the-art FVD score of 66.32 and Inception Score of 87.68 in
class-conditional video generation on UCF-101 256^2, surpassing recent
methods by more than 100%. Then, we show that it can be rapidly fine-tuned from
a base 36times 64 low-resolution generator for high-resolution 64 times
288 times 512 text-to-video synthesis. To the best of our knowledge, our
model is the first diffusion-based architecture which is trained on such high
resolutions entirely end-to-end. Project webpage:
https://snap-research.github.io/hpdm.
A novel diffusion-based architecture achieves state-of-the-art results in high-resolution video synthesis through deep context fusion and adaptive computation.
diffusion models deep context fusion adaptive computation FVD score Inception Score class-conditional video generation UCF-101 text-to-video synthesis end-to-end training