HyCodePolicy: Hybrid Language Controllers for Multimodal Monitoring and
  Decision in Embodied Agents
Recent advances in multimodal large language models (MLLMs) have enabled
richer perceptual grounding for code policy generation in embodied agents.
However, most existing systems lack effective mechanisms to adaptively monitor
policy execution and repair codes during task completion. In this work, we
introduce HyCodePolicy, a hybrid language-based control framework that
systematically integrates code synthesis, geometric grounding, perceptual
monitoring, and iterative repair into a closed-loop programming cycle for
embodied agents. Technically, given a natural language instruction, our system
first decomposes it into subgoals and generates an initial executable program
grounded in object-centric geometric primitives. The program is then executed
in simulation, while a vision-language model (VLM) observes selected
checkpoints to detect and localize execution failures and infer failure
reasons. By fusing structured execution traces capturing program-level events
with VLM-based perceptual feedback, HyCodePolicy infers failure causes and
repairs programs. This hybrid dual feedback mechanism enables self-correcting
program synthesis with minimal human supervision. Our results demonstrate that
HyCodePolicy significantly improves the robustness and sample efficiency of
robot manipulation policies, offering a scalable strategy for integrating
multimodal reasoning into autonomous decision-making pipelines.
68938603741a16f544fbcd9c
Yibin Liu
68938603741a16f544fbcd9d
Zhixuan Liang
68938603741a16f544fbcd9e
Zanxin Chen
68938603741a16f544fbcd9f
Tianxing Chen
68938603741a16f544fbcda0
Mengkang Hu
68938603741a16f544fbcda1
Wanxi Dong
68938603741a16f544fbcda2
Congsheng Xu
68938603741a16f544fbcda3
Zhaoming Han
68938603741a16f544fbcda4
Yusen Qin
68938603741a16f544fbcda5
Yao Mu
HyCodePolicy: Hybrid Language Controllers for Multimodal Monitoring and
  Decision in Embodied Agents
Recent advances in multimodal large language models (MLLMs) have enabled
richer perceptual grounding for code policy generation in embodied agents.
However, most existing systems lack effective mechanisms to adaptively monitor
policy execution and repair codes during task completion. In this work, we
introduce HyCodePolicy, a hybrid language-based control framework that
systematically integrates code synthesis, geometric grounding, perceptual
monitoring, and iterative repair into a closed-loop programming cycle for
embodied agents. Technically, given a natural language instruction, our system
first decomposes it into subgoals and generates an initial executable program
grounded in object-centric geometric primitives. The program is then executed
in simulation, while a vision-language model (VLM) observes selected
checkpoints to detect and localize execution failures and infer failure
reasons. By fusing structured execution traces capturing program-level events
with VLM-based perceptual feedback, HyCodePolicy infers failure causes and
repairs programs. This hybrid dual feedback mechanism enables self-correcting
program synthesis with minimal human supervision. Our results demonstrate that
HyCodePolicy significantly improves the robustness and sample efficiency of
robot manipulation policies, offering a scalable strategy for integrating
multimodal reasoning into autonomous decision-making pipelines.
HyCodePolicy integrates code synthesis, geometric grounding, perceptual monitoring, and iterative repair to enhance the robustness and efficiency of embodied agent policies.
multimodal large language models MLLMs code policy generation embodied agents HyCodePolicy code synthesis geometric grounding perceptual monitoring iterative repair closed-loop programming cycle natural language instruction subgoals executable program object-centric geometric primitives vision-language model VLM execution failures failure reasons structured execution traces perceptual feedback self-correcting program synthesis sample efficiency robot manipulation policies multimodal reasoning autonomous decision-making pipelines