AudioToken: Adaptation of Text-Conditioned Diffusion Models for
  Audio-to-Image Generation
In recent years, image generation has shown a great leap in performance,
where diffusion models play a central role. Although generating high-quality
images, such models are mainly conditioned on textual descriptions. This begs
the question: "how can we adopt such models to be conditioned on other
modalities?". In this paper, we propose a novel method utilizing latent
diffusion models trained for text-to-image-generation to generate images
conditioned on audio recordings. Using a pre-trained audio encoding model, the
proposed method encodes audio into a new token, which can be considered as an
adaptation layer between the audio and text representations. Such a modeling
paradigm requires a small number of trainable parameters, making the proposed
approach appealing for lightweight optimization. Results suggest the proposed
method is superior to the evaluated baseline methods, considering objective and
subjective metrics. Code and samples are available at:
https://pages.cs.huji.ac.il/adiyoss-lab/AudioToken.
646c12883e2a7b06594ad119
Guy Yariv
claimed_verified
2023-05-24T15:30:50.778Z
646d239f4220471ca0c6471c
https://cdn-avatars.huggingface.co/v1/production/uploads/646d239f4220471ca0c6471c/sRwzko8XEUVCkeD7jXceH.jpeg
Guy Yariv
user
GuyYariv
646c12883e2a7b06594ad11a
Itai Gat
claimed_verified
2023-05-23T08:11:32.894Z
62a8fa984d933c74bf410c16
/avatars/73519deba3176be9c23d49f749aee5da.svg
Itai Gat
user
itaigat
646c12883e2a7b06594ad11b
Lior Wolf
admin_assigned
2023-05-23T08:43:14.124Z
62f9f170363251ee40a08046
/avatars/67ad459db44b605be0c1f6bd33363d6e.svg
Lior Wolf
user
liorwolf
646c12883e2a7b06594ad11c
Yossi Adi
646c12883e2a7b06594ad11d
Idan Schwartz
claimed_verified
2023-05-23T12:22:52.549Z
617fb19badaa26f5e57e97cd
https://cdn-avatars.huggingface.co/v1/production/uploads/1645002335184-617fb19badaa26f5e57e97cd.png
Schwartz
user
Idan
AudioToken: Adaptation of Text-Conditioned Diffusion Models for
  Audio-to-Image Generation
In recent years, image generation has shown a great leap in performance,
where diffusion models play a central role. Although generating high-quality
images, such models are mainly conditioned on textual descriptions. This begs
the question: "how can we adopt such models to be conditioned on other
modalities?". In this paper, we propose a novel method utilizing latent
diffusion models trained for text-to-image-generation to generate images
conditioned on audio recordings. Using a pre-trained audio encoding model, the
proposed method encodes audio into a new token, which can be considered as an
adaptation layer between the audio and text representations. Such a modeling
paradigm requires a small number of trainable parameters, making the proposed
approach appealing for lightweight optimization. Results suggest the proposed
method is superior to the evaluated baseline methods, considering objective and
subjective metrics. Code and samples are available at:
https://pages.cs.huji.ac.il/adiyoss-lab/AudioToken.
A novel method using latent diffusion models and an audio encoding model generates images from audio recordings, outperforming baseline methods in both objective and subjective metrics.
latent diffusion models audio encoding model token lightweight optimization