AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and
  Video Generation
Recent Diffusion Transformers (DiTs) have shown impressive capabilities in
generating high-quality single-modality content, including images, videos, and
audio. However, it is still under-explored whether the transformer-based
diffuser can efficiently denoise the Gaussian noises towards superb multimodal
content creation. To bridge this gap, we introduce AV-DiT, a novel and
efficient audio-visual diffusion transformer designed to generate high-quality,
realistic videos with both visual and audio tracks. To minimize model
complexity and computational costs, AV-DiT utilizes a shared DiT backbone
pre-trained on image-only data, with only lightweight, newly inserted adapters
being trainable. This shared backbone facilitates both audio and video
generation. Specifically, the video branch incorporates a trainable temporal
attention layer into a frozen pre-trained DiT block for temporal consistency.
Additionally, a small number of trainable parameters adapt the image-based DiT
block for audio generation. An extra shared DiT block, equipped with
lightweight parameters, facilitates feature interaction between audio and
visual modalities, ensuring alignment. Extensive experiments on the AIST++ and
Landscape datasets demonstrate that AV-DiT achieves state-of-the-art
performance in joint audio-visual generation with significantly fewer tunable
parameters. Furthermore, our results highlight that a single shared image
generative backbone with modality-specific adaptations is sufficient for
constructing a joint audio-video generator. Our source code and pre-trained
models will be released.
666a52b2cf27eed0f20fbf5a
Kai Wang
666a52b2cf27eed0f20fbf5b
Shijian Deng
claimed_verified
2024-07-08T20:01:02.128Z
668bafa02c3897266dcbacd0
/avatars/ef41cd4a855ebbe6efc79f34c38ef136.svg
Shijian Deng
user
ShijianDeng
666a52b2cf27eed0f20fbf5c
Jing Shi
666a52b2cf27eed0f20fbf5d
Dimitrios Hatzinakos
666a52b2cf27eed0f20fbf5e
Yapeng Tian
AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and
  Video Generation
Recent Diffusion Transformers (DiTs) have shown impressive capabilities in
generating high-quality single-modality content, including images, videos, and
audio. However, it is still under-explored whether the transformer-based
diffuser can efficiently denoise the Gaussian noises towards superb multimodal
content creation. To bridge this gap, we introduce AV-DiT, a novel and
efficient audio-visual diffusion transformer designed to generate high-quality,
realistic videos with both visual and audio tracks. To minimize model
complexity and computational costs, AV-DiT utilizes a shared DiT backbone
pre-trained on image-only data, with only lightweight, newly inserted adapters
being trainable. This shared backbone facilitates both audio and video
generation. Specifically, the video branch incorporates a trainable temporal
attention layer into a frozen pre-trained DiT block for temporal consistency.
Additionally, a small number of trainable parameters adapt the image-based DiT
block for audio generation. An extra shared DiT block, equipped with
lightweight parameters, facilitates feature interaction between audio and
visual modalities, ensuring alignment. Extensive experiments on the AIST++ and
Landscape datasets demonstrate that AV-DiT achieves state-of-the-art
performance in joint audio-visual generation with significantly fewer tunable
parameters. Furthermore, our results highlight that a single shared image
generative backbone with modality-specific adaptations is sufficient for
constructing a joint audio-video generator. Our source code and pre-trained
models will be released.
AV-DiT, an efficient audio-visual diffusion transformer, generates high-quality joint audio-video content using a shared pretrained DiT backbone with modality-specific lightweight adaptations.
Diffusion Transformers DiTs Gaussian noises multimodal content creation audio-visual diffusion transformer pretrained DiT backbone lightweight adapters temporal attention layer feature interaction AIST++ Landscape datasets joint audio-visual generation