Seeing Voices: Generating A-Roll Video from Audio with Mirage
From professional filmmaking to user-generated content, creators and
consumers have long recognized that the power of video depends on the
harmonious integration of what we hear (the video's audio track) with what we
see (the video's image sequence). Current approaches to video generation either
ignore sound to focus on general-purpose but silent image sequence generation
or address both visual and audio elements but focus on restricted application
domains such as re-dubbing. We introduce Mirage, an audio-to-video foundation
model that excels at generating realistic, expressive output imagery from
scratch given an audio input. When integrated with existing methods for speech
synthesis (text-to-speech, or TTS), Mirage results in compelling multimodal
video. When trained on audio-video footage of people talking (A-roll) and
conditioned on audio containing speech, Mirage generates video of people
delivering a believable interpretation of the performance implicit in input
audio. Our central technical contribution is a unified method for training
self-attention-based audio-to-video generation models, either from scratch or
given existing weights. This methodology allows Mirage to retain generality as
an approach to audio-to-video generation while producing outputs of superior
subjective quality to methods that incorporate audio-specific architectures or
loss components specific to people, speech, or details of how images or audio
are captured. We encourage readers to watch and listen to the results of Mirage
for themselves (see paper and comments for links).
6849948548021d7f53998b64
Aditi Sundararaman
claimed_verified
2025-06-11T20:21:16.802Z
6849ab862603ba46f81d7d73
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/02xo4XQJx-1KxEcmughku.jpeg
aditi sundararaman
user
n0taditi
6849948548021d7f53998b65
Amogh Adishesha
6849948548021d7f53998b66
Andrew Jaegle
6849948548021d7f53998b67
Dan Bigioi
6849948548021d7f53998b68
Hyoung-Kyu Song
claimed_verified
2025-06-11T16:01:41.363Z
61f44bab7eba274ea80b74ce
https://cdn-avatars.huggingface.co/v1/production/uploads/61f44bab7eba274ea80b74ce/BRbKX1jephdZ7D44FATl4.jpeg
Hyoung-Kyu Song
user
deepkyu
6849948548021d7f53998b69
Jon Kyl
claimed_verified
2025-06-11T20:21:15.027Z
65b848353112ff9a78fdc940
/avatars/609576c2145316e225fe8066a76bde2d.svg
Jon Kyl
user
jon-kyl
6849948548021d7f53998b6a
Justin Mao
6849948548021d7f53998b6b
Kevin Lan
claimed_verified
2025-06-12T12:42:28.125Z
684a3d6a35cf4b6b0659c1b2
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/R2E0arkwBD2DPxa9f542A.png
Kevin Lan
user
kevin-lan
6849948548021d7f53998b6c
Mojtaba Komeili
6849948548021d7f53998b6d
ShahRukh Athar
claimed_verified
2025-06-12T12:42:25.917Z
650c3a2bd439bbbbadf40436
/avatars/a5a7c04a03580ae59c28eb6f90d43b47.svg
ShahRukh Athar
user
Sathar
6849948548021d7f53998b6e
Sheila Babayan
6849948548021d7f53998b6f
Stanislau Beliasau
6849948548021d7f53998b70
William Buchwalter
Seeing Voices: Generating A-Roll Video from Audio with Mirage
From professional filmmaking to user-generated content, creators and
consumers have long recognized that the power of video depends on the
harmonious integration of what we hear (the video's audio track) with what we
see (the video's image sequence). Current approaches to video generation either
ignore sound to focus on general-purpose but silent image sequence generation
or address both visual and audio elements but focus on restricted application
domains such as re-dubbing. We introduce Mirage, an audio-to-video foundation
model that excels at generating realistic, expressive output imagery from
scratch given an audio input. When integrated with existing methods for speech
synthesis (text-to-speech, or TTS), Mirage results in compelling multimodal
video. When trained on audio-video footage of people talking (A-roll) and
conditioned on audio containing speech, Mirage generates video of people
delivering a believable interpretation of the performance implicit in input
audio. Our central technical contribution is a unified method for training
self-attention-based audio-to-video generation models, either from scratch or
given existing weights. This methodology allows Mirage to retain generality as
an approach to audio-to-video generation while producing outputs of superior
subjective quality to methods that incorporate audio-specific architectures or
loss components specific to people, speech, or details of how images or audio
are captured. We encourage readers to watch and listen to the results of Mirage
for themselves (see paper and comments for links).
Mirage generates realistic video from audio inputs, integrating with speech synthesis to create compelling multimodal content through a unified, self-attention-based training approach.
audio-to-video foundation model self-attention-based audio-to-video generation models audio-video footage text-to-speech TTS A-roll