Multimodal Policy Internalization for Conversational Agents
Modern conversational agents like ChatGPT and Alexa+ rely on predefined
policies specifying metadata, response styles, and tool-usage rules. As these
LLM-based systems expand to support diverse business and user queries, such
policies, often implemented as in-context prompts, are becoming increasingly
complex and lengthy, making faithful adherence difficult and imposing large
fixed computational costs. With the rise of multimodal agents, policies that
govern visual and multimodal behaviors are critical but remain understudied.
Prior prompt-compression work mainly shortens task templates and
demonstrations, while existing policy-alignment studies focus only on
text-based safety rules. We introduce Multimodal Policy Internalization (MPI),
a new task that internalizes reasoning-intensive multimodal policies into model
parameters, enabling stronger policy-following without including the policy
during inference. MPI poses unique data and algorithmic challenges. We build
two datasets spanning synthetic and real-world decision-making and tool-using
tasks and propose TriMPI, a three-stage training framework. TriMPI first
injects policy knowledge via continual pretraining, then performs supervised
finetuning, and finally applies PolicyRollout, a GRPO-style reinforcement
learning extension that augments rollouts with policy-aware responses for
grounded exploration. TriMPI achieves notable gains in end-to-end accuracy,
generalization, and robustness to forgetting. As the first work on multimodal
policy internalization, we provide datasets, training recipes, and
comprehensive evaluations to foster future research. Project page:
https://mikewangwzhl.github.io/TriMPI.
68ee7ab17b0ac1ee3e599890
Zhenhailong Wang
68ee7ab17b0ac1ee3e599891
Jiateng Liu
68ee7ab17b0ac1ee3e599892
Amin Fazel
68ee7ab17b0ac1ee3e599893
Ritesh Sarkhel
68ee7ab17b0ac1ee3e599894
Xing Fan
68ee7ab17b0ac1ee3e599895
Xiang Li
68ee7ab17b0ac1ee3e599896
Chenlei Guo
68ee7ab17b0ac1ee3e599897
Heng Ji
68ee7ab17b0ac1ee3e599898
Ruhi Sarikaya
Multimodal Policy Internalization for Conversational Agents
Modern conversational agents like ChatGPT and Alexa+ rely on predefined
policies specifying metadata, response styles, and tool-usage rules. As these
LLM-based systems expand to support diverse business and user queries, such
policies, often implemented as in-context prompts, are becoming increasingly
complex and lengthy, making faithful adherence difficult and imposing large
fixed computational costs. With the rise of multimodal agents, policies that
govern visual and multimodal behaviors are critical but remain understudied.
Prior prompt-compression work mainly shortens task templates and
demonstrations, while existing policy-alignment studies focus only on
text-based safety rules. We introduce Multimodal Policy Internalization (MPI),
a new task that internalizes reasoning-intensive multimodal policies into model
parameters, enabling stronger policy-following without including the policy
during inference. MPI poses unique data and algorithmic challenges. We build
two datasets spanning synthetic and real-world decision-making and tool-using
tasks and propose TriMPI, a three-stage training framework. TriMPI first
injects policy knowledge via continual pretraining, then performs supervised
finetuning, and finally applies PolicyRollout, a GRPO-style reinforcement
learning extension that augments rollouts with policy-aware responses for
grounded exploration. TriMPI achieves notable gains in end-to-end accuracy,
generalization, and robustness to forgetting. As the first work on multimodal
policy internalization, we provide datasets, training recipes, and
comprehensive evaluations to foster future research. Project page:
https://mikewangwzhl.github.io/TriMPI.
Multimodal Policy Internalization (MPI) internalizes complex multimodal policies into model parameters, enhancing policy adherence and performance in conversational agents.
Multimodal Policy Internalization MPI TriMPI continual pretraining supervised finetuning PolicyRollout GRPO reinforcement learning end-to-end accuracy generalization robustness to forgetting