A Vision-Language-Action-Critic Model for Robotic Real-World
  Reinforcement Learning
Robotic real-world reinforcement learning (RL) with vision-language-action
(VLA) models is bottlenecked by sparse, handcrafted rewards and inefficient
exploration. We introduce VLAC, a general process reward model built upon
InternVL and trained on large scale heterogeneous datasets. Given pairwise
observations and a language goal, it outputs dense progress delta and done
signal, eliminating task-specific reward engineering, and supports one-shot
in-context transfer to unseen tasks and environments. VLAC is trained on
vision-language datasets to strengthen perception, dialogic and reasoning
capabilities, together with robot and human trajectories data that ground
action generation and progress estimation, and additionally strengthened to
reject irrelevant prompts as well as detect regression or stagnation by
constructing large numbers of negative and semantically mismatched samples.
With prompt control, a single VLAC model alternately generating reward and
action tokens, unifying critic and policy. Deployed inside an asynchronous
real-world RL loop, we layer a graded human-in-the-loop protocol (offline
demonstration replay, return and explore, human guided explore) that
accelerates exploration and stabilizes early learning. Across four distinct
real-world manipulation tasks, VLAC lifts success rates from about 30\% to
about 90\% within 200 real-world interaction episodes; incorporating
human-in-the-loop interventions yields a further 50% improvement in sample
efficiency and achieves up to 100% final success.
68d0b09d8adc5cd018d15abd
Shaopeng Zhai
68d0b09d8adc5cd018d15abe
Qi Zhang
68d0b09d8adc5cd018d15abf
Tianyi Zhang
68d0b09d8adc5cd018d15ac0
Fuxian Huang
admin_assigned
2025-09-22T03:02:51.610Z
6621c9e22b5271e44256538b
/avatars/59358fab5d17943937f9d1ca82cd2f3d.svg
huang
user
fuxian
68d0b09d8adc5cd018d15ac1
Haoran Zhang
admin_assigned
2025-09-22T03:02:40.943Z
647e5d1218274bce0306e8b5
/avatars/98266236ace7e56088f69af1d6c904db.svg
Haoran Zhang
user
haoranzhang
68d0b09d8adc5cd018d15ac2
Ming Zhou
68d0b09d8adc5cd018d15ac3
Shengzhe Zhang
admin_assigned
2025-09-22T03:02:32.543Z
64f0255350dae59768e973f6
/avatars/8708f8c41487e4d9a8b45df5bd36a926.svg
Shengzhe Zhang
user
andyzsz123
68d0b09d8adc5cd018d15ac4
Litao Liu
68d0b09d8adc5cd018d15ac5
Sixu Lin
admin_assigned
2025-09-22T03:02:11.587Z
68198bce0c88dfc744019d17
/avatars/79b040485072e7cd595d8448d2381271.svg
Sixu Lin
user
simonlin123
68d0b09d8adc5cd018d15ac6
Jiangmiao Pang
admin_assigned
2025-09-22T03:02:04.522Z
65783ee6ee33d547aecc3ffc
https://cdn-avatars.huggingface.co/v1/production/uploads/65783ee6ee33d547aecc3ffc/lWZX88c-0dCsN-yB9Jhlf.jpeg
Jiangmiao Pang
user
Jiangmiao
A Vision-Language-Action-Critic Model for Robotic Real-World
  Reinforcement Learning
Robotic real-world reinforcement learning (RL) with vision-language-action
(VLA) models is bottlenecked by sparse, handcrafted rewards and inefficient
exploration. We introduce VLAC, a general process reward model built upon
InternVL and trained on large scale heterogeneous datasets. Given pairwise
observations and a language goal, it outputs dense progress delta and done
signal, eliminating task-specific reward engineering, and supports one-shot
in-context transfer to unseen tasks and environments. VLAC is trained on
vision-language datasets to strengthen perception, dialogic and reasoning
capabilities, together with robot and human trajectories data that ground
action generation and progress estimation, and additionally strengthened to
reject irrelevant prompts as well as detect regression or stagnation by
constructing large numbers of negative and semantically mismatched samples.
With prompt control, a single VLAC model alternately generating reward and
action tokens, unifying critic and policy. Deployed inside an asynchronous
real-world RL loop, we layer a graded human-in-the-loop protocol (offline
demonstration replay, return and explore, human guided explore) that
accelerates exploration and stabilizes early learning. Across four distinct
real-world manipulation tasks, VLAC lifts success rates from about 30\% to
about 90\% within 200 real-world interaction episodes; incorporating
human-in-the-loop interventions yields a further 50% improvement in sample
efficiency and achieves up to 100% final success.
VLAC, a vision-language-action reward model, enhances real-world robotic reinforcement learning by providing dense rewards and enabling one-shot transfer, significantly improving success rates and sample efficiency.
reinforcement learning vision-language-action process reward model InternVL dense progress delta done signal task-specific reward engineering one-shot in-context transfer vision-language datasets robot and human trajectories negative samples semantically mismatched samples prompt control asynchronous real-world RL loop human-in-the-loop protocol offline demonstration replay return and explore human guided explore graded human-in-the-loop protocol