Code-as-Monitor: Constraint-aware Visual Programming for Reactive and
  Proactive Robotic Failure Detection
Automatic detection and prevention of open-set failures are crucial in
closed-loop robotic systems. Recent studies often struggle to simultaneously
identify unexpected failures reactively after they occur and prevent
foreseeable ones proactively. To this end, we propose Code-as-Monitor (CaM), a
novel paradigm leveraging the vision-language model (VLM) for both open-set
reactive and proactive failure detection. The core of our method is to
formulate both tasks as a unified set of spatio-temporal constraint
satisfaction problems and use VLM-generated code to evaluate them for real-time
monitoring. To enhance the accuracy and efficiency of monitoring, we further
introduce constraint elements that abstract constraint-related entities or
their parts into compact geometric elements. This approach offers greater
generality, simplifies tracking, and facilitates constraint-aware visual
programming by leveraging these elements as visual prompts. Experiments show
that CaM achieves a 28.7% higher success rate and reduces execution time by
31.8% under severe disturbances compared to baselines across three simulators
and a real-world setting. Moreover, CaM can be integrated with open-loop
control policies to form closed-loop systems, enabling long-horizon tasks in
cluttered scenes with dynamic environments.
6752744f8681347ff4207fc2
Enshen Zhou
claimed_verified
2024-12-06T09:08:56.444Z
63f08dc79cf89c9ed1bb89cd
/avatars/37290358ad00bbd752f519cfdec02f3e.svg
Zhoues
user
Zhoues
6752744f8681347ff4207fc3
Qi Su
6752744f8681347ff4207fc4
Cheng Chi
claimed_verified
2024-12-06T15:57:13.512Z
650bf938677f9e45963d672e
/avatars/7d4159067b5005a3a635e36b26b7b239.svg
Cheng Chi
user
ChuckChi
6752744f8681347ff4207fc5
Zhizheng Zhang
6752744f8681347ff4207fc6
Zhongyuan Wang
6752744f8681347ff4207fc7
Tiejun Huang
6752744f8681347ff4207fc8
Lu Sheng
claimed_verified
2024-12-06T15:56:27.453Z
65b722dbe02a17f0f8d1cc6b
/avatars/65f20601ef9b8ebfdddadd737f9153d6.svg
Lu Sheng
user
lsheng2024
6752744f8681347ff4207fc9
He Wang
Code-as-Monitor: Constraint-aware Visual Programming for Reactive and
  Proactive Robotic Failure Detection
Automatic detection and prevention of open-set failures are crucial in
closed-loop robotic systems. Recent studies often struggle to simultaneously
identify unexpected failures reactively after they occur and prevent
foreseeable ones proactively. To this end, we propose Code-as-Monitor (CaM), a
novel paradigm leveraging the vision-language model (VLM) for both open-set
reactive and proactive failure detection. The core of our method is to
formulate both tasks as a unified set of spatio-temporal constraint
satisfaction problems and use VLM-generated code to evaluate them for real-time
monitoring. To enhance the accuracy and efficiency of monitoring, we further
introduce constraint elements that abstract constraint-related entities or
their parts into compact geometric elements. This approach offers greater
generality, simplifies tracking, and facilitates constraint-aware visual
programming by leveraging these elements as visual prompts. Experiments show
that CaM achieves a 28.7% higher success rate and reduces execution time by
31.8% under severe disturbances compared to baselines across three simulators
and a real-world setting. Moreover, CaM can be integrated with open-loop
control policies to form closed-loop systems, enabling long-horizon tasks in
cluttered scenes with dynamic environments.
Code-as-Monitor (CaM) uses vision-language models to address both reactive and proactive open-set failure detection in closed-loop robotic systems by formulating them as spatio-temporal constraint satisfaction problems.
vision-language model VLM spatio-temporal constraint satisfaction problems code-as-monitor constraint elements geometric elements open-set failures closed-loop systems