LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action
  Models
Visual-Language-Action (VLA) models report impressive success rates on
robotic manipulation benchmarks, yet these results may mask fundamental
weaknesses in robustness. We perform a systematic vulnerability analysis by
introducing controlled perturbations across seven dimensions: objects layout,
camera viewpoints, robot initial states, language instructions, light
conditions, background textures and sensor noise. We comprehensively analyzed
multiple state-of-the-art models and revealed consistent brittleness beneath
apparent competence. Our analysis exposes critical weaknesses: models exhibit
extreme sensitivity to perturbation factors, including camera viewpoints and
robot initial states, with performance dropping from 95% to below 30% under
modest perturbations. Surprisingly, models are largely insensitive to language
variations, with further experiments revealing that models tend to ignore
language instructions completely. Our findings challenge the assumption that
high benchmark scores equate to true competency and highlight the need for
evaluation practices that assess reliability under realistic variation.
68f05beb36f8b025381e1995
Senyu Fei
68f05beb36f8b025381e1996
Siyin Wang
claimed_verified
2025-10-16T06:38:33.758Z
64c3c631e77ea9f28111172a
/avatars/495dbb73b69c399bae780da3118e332f.svg
Siyin Wang (SII)
user
sinwang
68f05beb36f8b025381e1997
Junhao Shi
68f05beb36f8b025381e1998
Zihao Dai
68f05beb36f8b025381e1999
Jikun Cai
68f05beb36f8b025381e199a
Pengfang Qian
68f05beb36f8b025381e199b
Li Ji
68f05beb36f8b025381e199c
Xinzhe He
68f05beb36f8b025381e199d
Shiduo Zhang
68f05beb36f8b025381e199e
Zhaoye Fei
68f05beb36f8b025381e199f
Jinlan Fu
68f05beb36f8b025381e19a0
Jingjing Gong
68f05beb36f8b025381e19a1
Xipeng Qiu
LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action
  Models
Visual-Language-Action (VLA) models report impressive success rates on
robotic manipulation benchmarks, yet these results may mask fundamental
weaknesses in robustness. We perform a systematic vulnerability analysis by
introducing controlled perturbations across seven dimensions: objects layout,
camera viewpoints, robot initial states, language instructions, light
conditions, background textures and sensor noise. We comprehensively analyzed
multiple state-of-the-art models and revealed consistent brittleness beneath
apparent competence. Our analysis exposes critical weaknesses: models exhibit
extreme sensitivity to perturbation factors, including camera viewpoints and
robot initial states, with performance dropping from 95% to below 30% under
modest perturbations. Surprisingly, models are largely insensitive to language
variations, with further experiments revealing that models tend to ignore
language instructions completely. Our findings challenge the assumption that
high benchmark scores equate to true competency and highlight the need for
evaluation practices that assess reliability under realistic variation.
State-of-the-art Visual-Language-Action models show high benchmark scores but are brittle to various perturbations, particularly in camera viewpoints and robot initial states, and often ignore language instructions.
Visual-Language-Action models vulnerability analysis perturbations objects layout camera viewpoints robot initial states language instructions light conditions background textures sensor noise brittleness reliability