InstructX: Towards Unified Visual Editing with MLLM Guidance
With recent advances in Multimodal Large Language Models (MLLMs) showing
strong visual understanding and reasoning, interest is growing in using them to
improve the editing performance of diffusion models. Despite rapid progress,
most studies lack an in-depth analysis of MLLM design choices. Moreover, the
integration of MLLMs and diffusion models remains an open challenge in some
difficult tasks, such as video editing. In this paper, we present InstructX, a
unified framework for image and video editing. Specifically, we conduct a
comprehensive study on integrating MLLMs and diffusion models for
instruction-driven editing across diverse tasks. Building on this study, we
analyze the cooperation and distinction between images and videos in unified
modeling. (1) We show that training on image data can lead to emergent video
editing capabilities without explicit supervision, thereby alleviating the
constraints imposed by scarce video training data. (2) By incorporating
modality-specific MLLM features, our approach effectively unifies image and
video editing tasks within a single model. Extensive experiments demonstrate
that our method can handle a broad range of image and video editing tasks and
achieves state-of-the-art performance.
68e88ae595e8e6771df38aea
Chong Mou
68e88ae595e8e6771df38aeb
Qichao Sun
claimed_verified
2025-10-11T13:46:03.152Z
674566cb79d6f3a9da7be0de
/avatars/b6a5384820e150405039aa2b9badac29.svg
Qichao Sun
user
Simons212
68e88ae595e8e6771df38aec
Yanze Wu
68e88ae595e8e6771df38aed
Pengze Zhang
68e88ae595e8e6771df38aee
Xinghui Li
68e88ae595e8e6771df38aef
Fulong Ye
68e88ae595e8e6771df38af0
Songtao Zhao
claimed_verified
2025-10-22T15:46:41.977Z
64ba142e5a29c19c8d261651
/avatars/d4ac943980fdab24b0e0c552eb527cda.svg
zhao
user
st0815
68e88ae595e8e6771df38af1
Qian He
InstructX: Towards Unified Visual Editing with MLLM Guidance
With recent advances in Multimodal Large Language Models (MLLMs) showing
strong visual understanding and reasoning, interest is growing in using them to
improve the editing performance of diffusion models. Despite rapid progress,
most studies lack an in-depth analysis of MLLM design choices. Moreover, the
integration of MLLMs and diffusion models remains an open challenge in some
difficult tasks, such as video editing. In this paper, we present InstructX, a
unified framework for image and video editing. Specifically, we conduct a
comprehensive study on integrating MLLMs and diffusion models for
instruction-driven editing across diverse tasks. Building on this study, we
analyze the cooperation and distinction between images and videos in unified
modeling. (1) We show that training on image data can lead to emergent video
editing capabilities without explicit supervision, thereby alleviating the
constraints imposed by scarce video training data. (2) By incorporating
modality-specific MLLM features, our approach effectively unifies image and
video editing tasks within a single model. Extensive experiments demonstrate
that our method can handle a broad range of image and video editing tasks and
achieves state-of-the-art performance.
InstructX integrates Multimodal Large Language Models and diffusion models for instruction-driven image and video editing, achieving state-of-the-art performance across diverse tasks.
Multimodal Large Language Models diffusion models instruction-driven editing video editing image editing unified modeling modality-specific features