Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for
  MLLMs
Large Language Models (LLMs) have shown remarkable success, and their
multimodal expansions (MLLMs) further unlock capabilities spanning images,
videos, and other modalities beyond text. However, despite this shift, prompt
optimization approaches, designed to reduce the burden of manual prompt
crafting while maximizing performance, remain confined to text, ultimately
limiting the full potential of MLLMs. Motivated by this gap, we introduce the
new problem of multimodal prompt optimization, which expands the prior
definition of prompt optimization to the multimodal space defined by the pairs
of textual and non-textual prompts. To tackle this problem, we then propose the
Multimodal Prompt Optimizer (MPO), a unified framework that not only performs
the joint optimization of multimodal prompts through alignment-preserving
updates but also guides the selection process of candidate prompts by
leveraging earlier evaluations as priors in a Bayesian-based selection
strategy. Through extensive experiments across diverse modalities that go
beyond text, such as images, videos, and even molecules, we demonstrate that
MPO outperforms leading text-only optimization methods, establishing multimodal
prompt optimization as a crucial step to realizing the potential of MLLMs.
68ec632fcd07fb414898c97b
Yumin Choi
claimed_verified
2025-10-13T10:05:53.909Z
64cfa0b9749587dbe01d0079
/avatars/93ca0a1d9c5578d052c5af0d4d1a0252.svg
Yumin Choi
user
YuminChoi
68ec632fcd07fb414898c97c
Dongki Kim
claimed_verified
2025-10-14T10:23:54.036Z
65633c5e84a9fbe322f87d81
/avatars/7233a555b43c669847a950ce5697c92c.svg
DongkiKim
user
DongkiKim
68ec632fcd07fb414898c97d
Jinheon Baek
68ec632fcd07fb414898c97e
Sung Ju Hwang
Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for
  MLLMs
Large Language Models (LLMs) have shown remarkable success, and their
multimodal expansions (MLLMs) further unlock capabilities spanning images,
videos, and other modalities beyond text. However, despite this shift, prompt
optimization approaches, designed to reduce the burden of manual prompt
crafting while maximizing performance, remain confined to text, ultimately
limiting the full potential of MLLMs. Motivated by this gap, we introduce the
new problem of multimodal prompt optimization, which expands the prior
definition of prompt optimization to the multimodal space defined by the pairs
of textual and non-textual prompts. To tackle this problem, we then propose the
Multimodal Prompt Optimizer (MPO), a unified framework that not only performs
the joint optimization of multimodal prompts through alignment-preserving
updates but also guides the selection process of candidate prompts by
leveraging earlier evaluations as priors in a Bayesian-based selection
strategy. Through extensive experiments across diverse modalities that go
beyond text, such as images, videos, and even molecules, we demonstrate that
MPO outperforms leading text-only optimization methods, establishing multimodal
prompt optimization as a crucial step to realizing the potential of MLLMs.
Multimodal Prompt Optimizer (MPO) extends prompt optimization to handle multiple data types, improving performance over text-only methods in various applications.
Large Language Models multimodal expansions prompt optimization Multimodal Prompt Optimizer alignment-preserving updates Bayesian-based selection strategy