SoundReactor: Frame-level Online Video-to-Audio Generation
Prevailing Video-to-Audio (V2A) generation models operate offline, assuming
an entire video sequence or chunks of frames are available beforehand. This
critically limits their use in interactive applications such as live content
creation and emerging generative world models. To address this gap, we
introduce the novel task of frame-level online V2A generation, where a model
autoregressively generates audio from video without access to future video
frames. Furthermore, we propose SoundReactor, which, to the best of our
knowledge, is the first simple yet effective framework explicitly tailored for
this task. Our design enforces end-to-end causality and targets low per-frame
latency with audio-visual synchronization. Our model's backbone is a
decoder-only causal transformer over continuous audio latents. For vision
conditioning, it leverages grid (patch) features extracted from the smallest
variant of the DINOv2 vision encoder, which are aggregated into a single token
per frame to maintain end-to-end causality and efficiency. The model is trained
through a diffusion pre-training followed by consistency fine-tuning to
accelerate the diffusion head decoding. On a benchmark of diverse gameplay
videos from AAA titles, our model successfully generates semantically and
temporally aligned, high-quality full-band stereo audio, validated by both
objective and human evaluations. Furthermore, our model achieves low per-frame
waveform-level latency (26.3ms with the head NFE=1, 31.5ms with NFE=4) on
30FPS, 480p videos using a single H100. Demo samples are available at
https://koichi-saito-sony.github.io/soundreactor/.
68e3359773e20ab577842018
Koichi Saito
claimed_verified
2025-10-07T12:28:28.387Z
665f7129b75fa1b1a705b7ce
/avatars/f9c8245c0441dc428edcefce8989444e.svg
Koichi Saito
user
koichisaito
68e3359773e20ab577842019
Julian Tanke
68e3359773e20ab57784201a
Christian Simon
68e3359773e20ab57784201b
Masato Ishii
68e3359773e20ab57784201c
Kazuki Shimada
68e3359773e20ab57784201d
Zachary Novack
68e3359773e20ab57784201e
Zhi Zhong
68e3359773e20ab57784201f
Akio Hayakawa
68e3359773e20ab577842020
Takashi Shibuya
68e3359773e20ab577842021
Yuki Mitsufuji
SoundReactor: Frame-level Online Video-to-Audio Generation
Prevailing Video-to-Audio (V2A) generation models operate offline, assuming
an entire video sequence or chunks of frames are available beforehand. This
critically limits their use in interactive applications such as live content
creation and emerging generative world models. To address this gap, we
introduce the novel task of frame-level online V2A generation, where a model
autoregressively generates audio from video without access to future video
frames. Furthermore, we propose SoundReactor, which, to the best of our
knowledge, is the first simple yet effective framework explicitly tailored for
this task. Our design enforces end-to-end causality and targets low per-frame
latency with audio-visual synchronization. Our model's backbone is a
decoder-only causal transformer over continuous audio latents. For vision
conditioning, it leverages grid (patch) features extracted from the smallest
variant of the DINOv2 vision encoder, which are aggregated into a single token
per frame to maintain end-to-end causality and efficiency. The model is trained
through a diffusion pre-training followed by consistency fine-tuning to
accelerate the diffusion head decoding. On a benchmark of diverse gameplay
videos from AAA titles, our model successfully generates semantically and
temporally aligned, high-quality full-band stereo audio, validated by both
objective and human evaluations. Furthermore, our model achieves low per-frame
waveform-level latency (26.3ms with the head NFE=1, 31.5ms with NFE=4) on
30FPS, 480p videos using a single H100. Demo samples are available at
https://koichi-saito-sony.github.io/soundreactor/.
A novel frame-level online Video-to-Audio generation model, SoundReactor, uses a causal transformer and DINOv2 vision encoder to generate high-quality, synchronized audio with low latency from video frames.
frame-level online V2A generation autoregressive generation causal transformer continuous audio latents DINOv2 vision encoder diffusion pre-training consistency fine-tuning audio-visual synchronization per-frame latency waveform-level latency