Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model
  Adaptation
We consider the task of generating diverse and realistic videos guided by
natural audio samples from a wide variety of semantic classes. For this task,
the videos are required to be aligned both globally and temporally with the
input audio: globally, the input audio is semantically associated with the
entire output video, and temporally, each segment of the input audio is
associated with a corresponding segment of that video. We utilize an existing
text-conditioned video generation model and a pre-trained audio encoder model.
The proposed method is based on a lightweight adaptor network, which learns to
map the audio-based representation to the input representation expected by the
text-to-video generation model. As such, it also enables video generation
conditioned on text, audio, and, for the first time as far as we can ascertain,
on both text and audio. We validate our method extensively on three datasets
demonstrating significant semantic diversity of audio-video samples and further
propose a novel evaluation metric (AV-Align) to assess the alignment of
generated videos with input audio samples. AV-Align is based on the detection
and comparison of energy peaks in both modalities. In comparison to recent
state-of-the-art approaches, our method generates videos that are better
aligned with the input sound, both with respect to content and temporal axis.
We also show that videos produced by our method present higher visual quality
and are more diverse.
65163fa0c661dc405bd8d841
Guy Yariv
claimed_verified
2023-09-29T07:04:21.968Z
646d239f4220471ca0c6471c
https://cdn-avatars.huggingface.co/v1/production/uploads/646d239f4220471ca0c6471c/sRwzko8XEUVCkeD7jXceH.jpeg
Guy Yariv
user
GuyYariv
65163fa0c661dc405bd8d842
Itai Gat
admin_assigned
2023-09-29T11:48:21.716Z
62a8fa984d933c74bf410c16
/avatars/73519deba3176be9c23d49f749aee5da.svg
Itai Gat
user
itaigat
65163fa0c661dc405bd8d843
Sagie Benaim
admin_assigned
2023-09-29T11:48:30.102Z
6345a9b9a8c2ff9f1377faab
/avatars/a5f2b999ef8b967b2af9f41afcd9d475.svg
Sagie Benaim
user
sagiebenaim
65163fa0c661dc405bd8d844
Lior Wolf
admin_assigned
2023-09-29T11:48:52.763Z
62f9f170363251ee40a08046
/avatars/67ad459db44b605be0c1f6bd33363d6e.svg
Lior Wolf
user
liorwolf
65163fa0c661dc405bd8d845
Idan Schwartz
65163fa0c661dc405bd8d846
Yossi Adi
admin_assigned
2023-09-29T11:49:11.717Z
6481e135578646b5c2386728
https://cdn-avatars.huggingface.co/v1/production/uploads/6481e135578646b5c2386728/SPva4iNw0pORiCXD45cx9.jpeg
Yossi Adi
user
adiyoss
Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model
  Adaptation
We consider the task of generating diverse and realistic videos guided by
natural audio samples from a wide variety of semantic classes. For this task,
the videos are required to be aligned both globally and temporally with the
input audio: globally, the input audio is semantically associated with the
entire output video, and temporally, each segment of the input audio is
associated with a corresponding segment of that video. We utilize an existing
text-conditioned video generation model and a pre-trained audio encoder model.
The proposed method is based on a lightweight adaptor network, which learns to
map the audio-based representation to the input representation expected by the
text-to-video generation model. As such, it also enables video generation
conditioned on text, audio, and, for the first time as far as we can ascertain,
on both text and audio. We validate our method extensively on three datasets
demonstrating significant semantic diversity of audio-video samples and further
propose a novel evaluation metric (AV-Align) to assess the alignment of
generated videos with input audio samples. AV-Align is based on the detection
and comparison of energy peaks in both modalities. In comparison to recent
state-of-the-art approaches, our method generates videos that are better
aligned with the input sound, both with respect to content and temporal axis.
We also show that videos produced by our method present higher visual quality
and are more diverse.
A lightweight adaptor network improves video generation conditioned on audio and text by aligning audio segments with video segments and enhancing alignment and diversity.
text-conditioned video generation model pre-trained audio encoder model lightweight adaptor network audio-based representation input representation text-to-video generation model AV-Align energy peaks detection and comparison audio-video samples semantic diversity visual quality