In-Context LoRA for Diffusion Transformers
Recent research arXiv:2410.15027 has explored the use of diffusion
transformers (DiTs) for task-agnostic image generation by simply concatenating
attention tokens across images. However, despite substantial computational
resources, the fidelity of the generated images remains suboptimal. In this
study, we reevaluate and streamline this framework by hypothesizing that
text-to-image DiTs inherently possess in-context generation capabilities,
requiring only minimal tuning to activate them. Through diverse task
experiments, we qualitatively demonstrate that existing text-to-image DiTs can
effectively perform in-context generation without any tuning. Building on this
insight, we propose a remarkably simple pipeline to leverage the in-context
abilities of DiTs: (1) concatenate images instead of tokens, (2) perform joint
captioning of multiple images, and (3) apply task-specific LoRA tuning using
small datasets (e.g., 20sim 100 samples) instead of full-parameter tuning
with large datasets. We name our models In-Context LoRA (IC-LoRA). This
approach requires no modifications to the original DiT models, only changes to
the training data. Remarkably, our pipeline generates high-fidelity image sets
that better adhere to prompts. While task-specific in terms of tuning data, our
framework remains task-agnostic in architecture and pipeline, offering a
powerful tool for the community and providing valuable insights for further
research on product-level task-agnostic generation systems. We release our
code, data, and models at https://github.com/ali-vilab/In-Context-LoRA
67246f26a2714fc915dd1104
Lianghua Huang
claimed_verified
2024-11-03T14:22:06.089Z
6522cf31777019ca30d85725
/avatars/a180b096e438e429d445b68fe703e43f.svg
Lianghua Huang
user
lhhuang
67246f26a2714fc915dd1105
Wei Wang
admin_assigned
2024-11-04T17:42:35.397Z
62fa0ffe0697d224219a0cb7
/avatars/f0ef59e1c0cf4ab4fe5cee08d488bd03.svg
Wei Wang
user
WeiWang
67246f26a2714fc915dd1106
Zhi-Fan Wu
admin_assigned
2024-11-04T17:42:28.526Z
64f0337e1dccbf71c3946204
/avatars/955d0533b4b7219773ef7b9a16c05848.svg
Wu Zhi-Fan
user
wuzhifan
67246f26a2714fc915dd1107
Yupeng Shi
admin_assigned
2024-11-04T17:42:14.408Z
6433edd99bd5a84b5dc8f5df
/avatars/ceaf87065ebaebabf1c0510185480ee9.svg
shi yupeng
user
shiyp123
67246f26a2714fc915dd1108
Huanzhang Dou
admin_assigned
2024-11-04T17:42:00.682Z
64c8ad3a527d763655600f6f
/avatars/6a4ab87fe040d666313f59d93712cf73.svg
Huanzhang Dou
user
Huanzhang
67246f26a2714fc915dd1109
Chen Liang
67246f26a2714fc915dd110a
Yutong Feng
admin_assigned
2024-11-04T17:10:51.356Z
64a54e468cfaa458bd6844bf
https://cdn-avatars.huggingface.co/v1/production/uploads/64a54e468cfaa458bd6844bf/5Gmf4tAr59GNl-2VaZDbu.png
Yutong Feng
user
fengyutong
67246f26a2714fc915dd110b
Yu Liu
67246f26a2714fc915dd110c
Jingren Zhou
admin_assigned
2024-11-04T17:10:44.913Z
602f88f5e8149a962412a667
/avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
Zhou
user
Jingren
In-Context LoRA for Diffusion Transformers
Recent research arXiv:2410.15027 has explored the use of diffusion
transformers (DiTs) for task-agnostic image generation by simply concatenating
attention tokens across images. However, despite substantial computational
resources, the fidelity of the generated images remains suboptimal. In this
study, we reevaluate and streamline this framework by hypothesizing that
text-to-image DiTs inherently possess in-context generation capabilities,
requiring only minimal tuning to activate them. Through diverse task
experiments, we qualitatively demonstrate that existing text-to-image DiTs can
effectively perform in-context generation without any tuning. Building on this
insight, we propose a remarkably simple pipeline to leverage the in-context
abilities of DiTs: (1) concatenate images instead of tokens, (2) perform joint
captioning of multiple images, and (3) apply task-specific LoRA tuning using
small datasets (e.g., 20sim 100 samples) instead of full-parameter tuning
with large datasets. We name our models In-Context LoRA (IC-LoRA). This
approach requires no modifications to the original DiT models, only changes to
the training data. Remarkably, our pipeline generates high-fidelity image sets
that better adhere to prompts. While task-specific in terms of tuning data, our
framework remains task-agnostic in architecture and pipeline, offering a
powerful tool for the community and providing valuable insights for further
research on product-level task-agnostic generation systems. We release our
code, data, and models at https://github.com/ali-vilab/In-Context-LoRA
Researchers propose IC-LoRA, a simple pipeline that leverages the in-context generation capabilities of diffusion transformers without requiring extensive tuning, resulting in high-fidelity image generation that adheres to prompts.
diffusion transformers DiTs attention tokens in-context generation joint captioning LoRA tuning task-specific LoRA