DreamOmni2: Multimodal Instruction-based Editing and Generation
Recent advancements in instruction-based image editing and subject-driven
generation have garnered significant attention, yet both tasks still face
limitations in meeting practical user needs. Instruction-based editing relies
solely on language instructions, which often fail to capture specific editing
details, making reference images necessary. Meanwhile, subject-driven
generation is limited to combining concrete objects or people, overlooking
broader, abstract concepts. To address these challenges, we propose two novel
tasks: multimodal instruction-based editing and generation. These tasks support
both text and image instructions and extend the scope to include both concrete
and abstract concepts, greatly enhancing their practical applications. We
introduce DreamOmni2, tackling two primary challenges: data creation and model
framework design. Our data synthesis pipeline consists of three steps: (1)
using a feature mixing method to create extraction data for both abstract and
concrete concepts, (2) generating multimodal instruction-based editing training
data using the editing and extraction models, and (3) further applying the
extraction model to create training data for multimodal instruction-based
editing. For the framework, to handle multi-image input, we propose an index
encoding and position encoding shift scheme, which helps the model distinguish
images and avoid pixel confusion. Additionally, we introduce joint training
with the VLM and our generation/editing model to better process complex
instructions. In addition, we have proposed comprehensive benchmarks for these
two new tasks to drive their development. Experiments show that DreamOmni2 has
achieved impressive results. Models and codes will be released.
68e949cbb12ca14cb8a1660e
Bin Xia
68e949cbb12ca14cb8a1660f
Bohao Peng
68e949cbb12ca14cb8a16610
Yuechen Zhang
68e949cbb12ca14cb8a16611
Junjia Huang
68e949cbb12ca14cb8a16612
Jiyang Liu
68e949cbb12ca14cb8a16613
Jingyao Li
68e949cbb12ca14cb8a16614
Haoru Tan
68e949cbb12ca14cb8a16615
Sitong Wu
68e949cbb12ca14cb8a16616
Chengyao Wang
claimed_verified
2025-10-11T13:45:00.583Z
6423e35b30b0e4ab36dd1b16
https://cdn-avatars.huggingface.co/v1/production/uploads/6423e35b30b0e4ab36dd1b16/pea6LVDS9PQAxvQt9GUiZ.jpeg
Wang Chengyao
user
wcy1122
68e949cbb12ca14cb8a16617
Yitong Wang
68e949cbb12ca14cb8a16618
Xinglong Wu
68e949cbb12ca14cb8a16619
Bei Yu
68e949cbb12ca14cb8a1661a
Jiaya Jia
DreamOmni2: Multimodal Instruction-based Editing and Generation
Recent advancements in instruction-based image editing and subject-driven
generation have garnered significant attention, yet both tasks still face
limitations in meeting practical user needs. Instruction-based editing relies
solely on language instructions, which often fail to capture specific editing
details, making reference images necessary. Meanwhile, subject-driven
generation is limited to combining concrete objects or people, overlooking
broader, abstract concepts. To address these challenges, we propose two novel
tasks: multimodal instruction-based editing and generation. These tasks support
both text and image instructions and extend the scope to include both concrete
and abstract concepts, greatly enhancing their practical applications. We
introduce DreamOmni2, tackling two primary challenges: data creation and model
framework design. Our data synthesis pipeline consists of three steps: (1)
using a feature mixing method to create extraction data for both abstract and
concrete concepts, (2) generating multimodal instruction-based editing training
data using the editing and extraction models, and (3) further applying the
extraction model to create training data for multimodal instruction-based
editing. For the framework, to handle multi-image input, we propose an index
encoding and position encoding shift scheme, which helps the model distinguish
images and avoid pixel confusion. Additionally, we introduce joint training
with the VLM and our generation/editing model to better process complex
instructions. In addition, we have proposed comprehensive benchmarks for these
two new tasks to drive their development. Experiments show that DreamOmni2 has
achieved impressive results. Models and codes will be released.
DreamOmni2 addresses limitations in instruction-based image editing and subject-driven generation by introducing multimodal instruction-based editing and generation tasks, utilizing feature mixing, index encoding, and joint training with a VLM.
instruction-based editing subject-driven generation multimodal instruction-based editing feature mixing index encoding position encoding joint training VLM