Video Object Segmentation-Aware Audio Generation
Existing multimodal audio generation models often lack precise user control,
which limits their applicability in professional Foley workflows. In
particular, these models focus on the entire video and do not provide precise
methods for prioritizing a specific object within a scene, generating
unnecessary background sounds, or focusing on the wrong objects. To address
this gap, we introduce the novel task of video object segmentation-aware audio
generation, which explicitly conditions sound synthesis on object-level
segmentation maps. We present SAGANet, a new multimodal generative model that
enables controllable audio generation by leveraging visual segmentation masks
along with video and textual cues. Our model provides users with fine-grained
and visually localized control over audio generation. To support this task and
further research on segmentation-aware Foley, we propose Segmented Music Solos,
a benchmark dataset of musical instrument performance videos with segmentation
information. Our method demonstrates substantial improvements over current
state-of-the-art methods and sets a new standard for controllable,
high-fidelity Foley synthesis. Code, samples, and Segmented Music Solos are
available at https://saganet.notion.site
68dd2684e795697646c0a14e
Ilpo Viertola
claimed_verified
2025-10-01T15:28:18.336Z
64b7c3d5f72c6cd946dff515
/avatars/f579bc83b529782bbe6109f18d89837a.svg
Ilpo Viertola
user
bilpo
68dd2684e795697646c0a14f
Vladimir Iashin
68dd2684e795697646c0a150
Esa Rahtu
Video Object Segmentation-Aware Audio Generation
Existing multimodal audio generation models often lack precise user control,
which limits their applicability in professional Foley workflows. In
particular, these models focus on the entire video and do not provide precise
methods for prioritizing a specific object within a scene, generating
unnecessary background sounds, or focusing on the wrong objects. To address
this gap, we introduce the novel task of video object segmentation-aware audio
generation, which explicitly conditions sound synthesis on object-level
segmentation maps. We present SAGANet, a new multimodal generative model that
enables controllable audio generation by leveraging visual segmentation masks
along with video and textual cues. Our model provides users with fine-grained
and visually localized control over audio generation. To support this task and
further research on segmentation-aware Foley, we propose Segmented Music Solos,
a benchmark dataset of musical instrument performance videos with segmentation
information. Our method demonstrates substantial improvements over current
state-of-the-art methods and sets a new standard for controllable,
high-fidelity Foley synthesis. Code, samples, and Segmented Music Solos are
available at https://saganet.notion.site
SAGANet, a multimodal generative model, enhances audio generation by using object-level segmentation maps, improving control and fidelity in professional Foley workflows.
multimodal audio generation video object segmentation-aware audio generation SAGANet sound synthesis visual segmentation masks Segmented Music Solos Foley synthesis