Sounding that Object: Interactive Object-Aware Image to Audio Generation
Generating accurate sounds for complex audio-visual scenes is challenging,
especially in the presence of multiple objects and sound sources. In this
paper, we propose an {\em interactive object-aware audio generation} model that
grounds sound generation in user-selected visual objects within images. Our
method integrates object-centric learning into a conditional latent diffusion
model, which learns to associate image regions with their corresponding sounds
through multi-modal attention. At test time, our model employs image
segmentation to allow users to interactively generate sounds at the {\em
object} level. We theoretically validate that our attention mechanism
functionally approximates test-time segmentation masks, ensuring the generated
audio aligns with selected objects. Quantitative and qualitative evaluations
show that our model outperforms baselines, achieving better alignment between
objects and their associated sounds. Project page:
https://tinglok.netlify.app/files/avobject/
6841b0a2d198c64464545ae9
Tingle Li
6841b0a2d198c64464545aea
Baihe Huang
6841b0a2d198c64464545aeb
Xiaobin Zhuang
claimed_verified
2025-06-05T15:07:43.456Z
63774ca43a63a2983ffc12f9
https://cdn-avatars.huggingface.co/v1/production/uploads/63774ca43a63a2983ffc12f9/jUJasGf5_2rwiEun_eYl7.png
xiaobin zhuang
user
xiaobinzhuang
6841b0a2d198c64464545aec
Dongya Jia
6841b0a2d198c64464545aed
Jiawei Chen
6841b0a2d198c64464545aee
Yuping Wang
6841b0a2d198c64464545aef
Zhuo Chen
6841b0a2d198c64464545af0
Gopala Anumanchipalli
6841b0a2d198c64464545af1
Yuxuan Wang
Sounding that Object: Interactive Object-Aware Image to Audio Generation
Generating accurate sounds for complex audio-visual scenes is challenging,
especially in the presence of multiple objects and sound sources. In this
paper, we propose an {\em interactive object-aware audio generation} model that
grounds sound generation in user-selected visual objects within images. Our
method integrates object-centric learning into a conditional latent diffusion
model, which learns to associate image regions with their corresponding sounds
through multi-modal attention. At test time, our model employs image
segmentation to allow users to interactively generate sounds at the {\em
object} level. We theoretically validate that our attention mechanism
functionally approximates test-time segmentation masks, ensuring the generated
audio aligns with selected objects. Quantitative and qualitative evaluations
show that our model outperforms baselines, achieving better alignment between
objects and their associated sounds. Project page:
https://tinglok.netlify.app/files/avobject/
An interactive object-aware audio generation model aligns sounds with user-selected visual objects in images using a conditional latent diffusion model with multi-modal attention.
interactive object-aware audio generation object-centric learning conditional latent diffusion model multi-modal attention image segmentation quantitative evaluation qualitative evaluation