ROICtrl: Boosting Instance Control for Visual Generation
Natural language often struggles to accurately associate positional and
attribute information with multiple instances, which limits current text-based
visual generation models to simpler compositions featuring only a few dominant
instances. To address this limitation, this work enhances diffusion models by
introducing regional instance control, where each instance is governed by a
bounding box paired with a free-form caption. Previous methods in this area
typically rely on implicit position encoding or explicit attention masks to
separate regions of interest (ROIs), resulting in either inaccurate coordinate
injection or large computational overhead. Inspired by ROI-Align in object
detection, we introduce a complementary operation called ROI-Unpool. Together,
ROI-Align and ROI-Unpool enable explicit, efficient, and accurate ROI
manipulation on high-resolution feature maps for visual generation. Building on
ROI-Unpool, we propose ROICtrl, an adapter for pretrained diffusion models that
enables precise regional instance control. ROICtrl is compatible with
community-finetuned diffusion models, as well as with existing spatial-based
add-ons (\eg, ControlNet, T2I-Adapter) and embedding-based add-ons (\eg,
IP-Adapter, ED-LoRA), extending their applications to multi-instance
generation. Experiments show that ROICtrl achieves superior performance in
regional instance control while significantly reducing computational costs.
6747cff9cf8e0c077f37d066
Yuchao Gu
admin_assigned
2024-11-28T09:54:59.262Z
63021630a35b21bd8a53305a
/avatars/7a7e8b39749eda61e57d8a1908726558.svg
Gu Yuchao
user
guyuchao
6747cff9cf8e0c077f37d067
Yipin Zhou
6747cff9cf8e0c077f37d068
Yunfan Ye
admin_assigned
2024-11-28T09:55:39.519Z
63b8c5ec3d5ccb207f57f03a
/avatars/070f535e946b5cadb82f61e5ca0a90be.svg
Yunfan Ye
user
365sleep
6747cff9cf8e0c077f37d069
Yixin Nie
claimed_verified
2024-11-28T09:19:47.282Z
5f7919d5f1e7ef6e919a1f55
/avatars/fb747fb43b65ccdfb08d7bf71525f92f.svg
Yixin Nie
user
ynie
6747cff9cf8e0c077f37d06a
Licheng Yu
6747cff9cf8e0c077f37d06b
Pingchuan Ma
claimed_verified
2024-11-28T09:19:45.738Z
64c163ecd9bdb0d64d58f716
https://cdn-avatars.huggingface.co/v1/production/uploads/64c163ecd9bdb0d64d58f716/ln8536t4XH4ZUYxdSmlId.png
Pingchuan Ma
user
pcma
6747cff9cf8e0c077f37d06c
Kevin Qinghong Lin
admin_assigned
2024-11-28T09:54:20.686Z
64440be5af034cdfd69ca3a7
https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
Qinghong (Kevin) Lin
user
KevinQHLin
6747cff9cf8e0c077f37d06d
Mike Zheng Shou
admin_assigned
2024-11-28T09:54:09.470Z
661ab3da2b14565c7acccf5c
/avatars/fa4fc03664803e02aede4d4c3d50b393.svg
Mike Zheng Shou
user
AnalMom
ROICtrl: Boosting Instance Control for Visual Generation
Natural language often struggles to accurately associate positional and
attribute information with multiple instances, which limits current text-based
visual generation models to simpler compositions featuring only a few dominant
instances. To address this limitation, this work enhances diffusion models by
introducing regional instance control, where each instance is governed by a
bounding box paired with a free-form caption. Previous methods in this area
typically rely on implicit position encoding or explicit attention masks to
separate regions of interest (ROIs), resulting in either inaccurate coordinate
injection or large computational overhead. Inspired by ROI-Align in object
detection, we introduce a complementary operation called ROI-Unpool. Together,
ROI-Align and ROI-Unpool enable explicit, efficient, and accurate ROI
manipulation on high-resolution feature maps for visual generation. Building on
ROI-Unpool, we propose ROICtrl, an adapter for pretrained diffusion models that
enables precise regional instance control. ROICtrl is compatible with
community-finetuned diffusion models, as well as with existing spatial-based
add-ons (\eg, ControlNet, T2I-Adapter) and embedding-based add-ons (\eg,
IP-Adapter, ED-LoRA), extending their applications to multi-instance
generation. Experiments show that ROICtrl achieves superior performance in
regional instance control while significantly reducing computational costs.
ROICtrl, an adapter for diffusion models using ROI-Align and ROI-Unpool, improves regional instance control in text-based visual generation with lower computational costs.
diffusion models regional instance control bounding box free-form caption regions of interest implicit position encoding explicit attention masks ROI-Align ROI-Unpool ROICtrl ControlNet T2I-Adapter IP-Adapter ED-LoRA multi-instance generation