Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion
As a common image editing operation, image composition involves integrating
foreground objects into background scenes. In this paper, we expand the
application of the concept of Affordance from human-centered image composition
tasks to a more general object-scene composition framework, addressing the
complex interplay between foreground objects and background scenes. Following
the principle of Affordance, we define the affordance-aware object insertion
task, which aims to seamlessly insert any object into any scene with various
position prompts. To address the limited data issue and incorporate this task,
we constructed the SAM-FB dataset, which contains over 3 million examples
across more than 3,000 object categories. Furthermore, we propose the
Mask-Aware Dual Diffusion (MADD) model, which utilizes a dual-stream
architecture to simultaneously denoise the RGB image and the insertion mask. By
explicitly modeling the insertion mask in the diffusion process, MADD
effectively facilitates the notion of affordance. Extensive experimental
results show that our method outperforms the state-of-the-art methods and
exhibits strong generalization performance on in-the-wild images. Please refer
to our code on https://github.com/KaKituken/affordance-aware-any.
6764e0339aeafaa0d8405e74
Jixuan He
admin_assigned
2024-12-20T09:24:47.947Z
64a5bda1733731f38378fdd3
/avatars/3ec7c553f92ced4ae0420ce16f7e71b9.svg
Jixuan HE
user
Kakituken
6764e0339aeafaa0d8405e75
Wanhua Li
claimed_verified
2024-12-20T08:32:44.737Z
658bb7e47459b6e471b9d2e6
/avatars/efd8051b468b4dbcb5d149479de67c58.svg
Wanhua Li
user
EthanTaylor
6764e0339aeafaa0d8405e76
Ye Liu
claimed_verified
2025-03-21T15:01:33.841Z
6250eb5c1fd03f78d0ae550f
https://cdn-avatars.huggingface.co/v1/production/uploads/1649470297585-noauth.jpeg
Ye Liu
user
yeliudev
6764e0339aeafaa0d8405e77
Junsik Kim
6764e0339aeafaa0d8405e78
Donglai Wei
admin_assigned
2024-12-20T09:25:00.011Z
628c2c8ab80bb09700d6cb1d
/avatars/384f75778fd5f07f249f2815a3039dca.svg
donglai wei
user
dwei
6764e0339aeafaa0d8405e79
Hanspeter Pfister
admin_assigned
2024-12-20T09:24:54.200Z
62acc69e36f7c7b7f65fccca
https://cdn-avatars.huggingface.co/v1/production/uploads/62acc69e36f7c7b7f65fccca/S8o0XE6TaQwLU8q3QPkct.png
Hanspeter Pfister
user
hpfister
Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion
As a common image editing operation, image composition involves integrating
foreground objects into background scenes. In this paper, we expand the
application of the concept of Affordance from human-centered image composition
tasks to a more general object-scene composition framework, addressing the
complex interplay between foreground objects and background scenes. Following
the principle of Affordance, we define the affordance-aware object insertion
task, which aims to seamlessly insert any object into any scene with various
position prompts. To address the limited data issue and incorporate this task,
we constructed the SAM-FB dataset, which contains over 3 million examples
across more than 3,000 object categories. Furthermore, we propose the
Mask-Aware Dual Diffusion (MADD) model, which utilizes a dual-stream
architecture to simultaneously denoise the RGB image and the insertion mask. By
explicitly modeling the insertion mask in the diffusion process, MADD
effectively facilitates the notion of affordance. Extensive experimental
results show that our method outperforms the state-of-the-art methods and
exhibits strong generalization performance on in-the-wild images. Please refer
to our code on https://github.com/KaKituken/affordance-aware-any.
The Mask-Aware Dual Diffusion (MADD) model enables seamless object insertion into scenes by explicitly modeling the insertion mask in the diffusion process, addressing data limitations and generalizing well to real-world images.
Affordance affordance-aware object insertion SAM-FB dataset Mask-Aware Dual Diffusion MADD model dual-stream architecture diffusion process