Enhancing Vision-Language Model Training with Reinforcement Learning in
  Synthetic Worlds for Real-World Success
Interactive multimodal agents must convert raw visual observations into
coherent sequences of language-conditioned actions -- a capability that current
vision-language models (VLMs) still lack. Earlier reinforcement-learning (RL)
efforts could, in principle, endow VLMs with such skills, but they have seldom
tested whether the learned behaviours generalize beyond their training
simulators, and they depend either on brittle hyperparameter tuning or on
dense-reward environments with low state variability. We introduce
Vision-Language Decoupled Actor-Critic (VL-DAC), a lightweight,
hyperparameter-free RL algorithm. VL-DAC applies PPO updates to action tokens
while learning value only at the environment-step level: an arrangement, to our
knowledge, not previously explored for large VLMs or LLMs. This simple
decoupling removes unstable weighting terms and yields faster, more reliable
convergence. Training a single VLM with VL-DAC in one inexpensive simulator at
a time (MiniWorld, Gym-Cards, ALFWorld, or WebShop) already produces policies
that generalize widely: +50\% relative on BALROG (game-centric agentic
control), +5\% relative on the hardest part of VSI-Bench (spatial planning),
and +2\% on VisualWebBench (web navigation), all without degrading general
image understanding accuracy. These results provide the first evidence that a
simple RL algorithm can train VLMs entirely in cheap synthetic worlds while
delivering measurable gains on real-image agentic, spatial-reasoning, and
web-navigation benchmarks.
68947269741a16f544fbd068
George Bredis
claimed_verified
2025-08-07T10:37:32.805Z
6348202122bc15d2636ccf87
https://cdn-avatars.huggingface.co/v1/production/uploads/1673510811337-6348202122bc15d2636ccf87.jpeg
Natyren
user
GeorgeBredis
68947269741a16f544fbd069
Stanislav Dereka
68947269741a16f544fbd06a
Viacheslav Sinii
68947269741a16f544fbd06b
Ruslan Rakhimov
claimed_verified
2025-08-19T10:50:19.322Z
6482ddba57d6e62bf4d8b837
https://cdn-avatars.huggingface.co/v1/production/uploads/6482ddba57d6e62bf4d8b837/XmiNnZ7BB3vgrvEV8nsNI.png
Ruslan Rakhimov
user
rusrakhimov
68947269741a16f544fbd06c
Daniil Gavrilov
claimed_verified
2025-08-07T10:37:30.344Z
62a9c8edc19f92ae443ab37f
https://cdn-avatars.huggingface.co/v1/production/uploads/1669110208492-62a9c8edc19f92ae443ab37f.png
Daniil Gavrilov
user
kefirski
Enhancing Vision-Language Model Training with Reinforcement Learning in
  Synthetic Worlds for Real-World Success
Interactive multimodal agents must convert raw visual observations into
coherent sequences of language-conditioned actions -- a capability that current
vision-language models (VLMs) still lack. Earlier reinforcement-learning (RL)
efforts could, in principle, endow VLMs with such skills, but they have seldom
tested whether the learned behaviours generalize beyond their training
simulators, and they depend either on brittle hyperparameter tuning or on
dense-reward environments with low state variability. We introduce
Vision-Language Decoupled Actor-Critic (VL-DAC), a lightweight,
hyperparameter-free RL algorithm. VL-DAC applies PPO updates to action tokens
while learning value only at the environment-step level: an arrangement, to our
knowledge, not previously explored for large VLMs or LLMs. This simple
decoupling removes unstable weighting terms and yields faster, more reliable
convergence. Training a single VLM with VL-DAC in one inexpensive simulator at
a time (MiniWorld, Gym-Cards, ALFWorld, or WebShop) already produces policies
that generalize widely: +50\% relative on BALROG (game-centric agentic
control), +5\% relative on the hardest part of VSI-Bench (spatial planning),
and +2\% on VisualWebBench (web navigation), all without degrading general
image understanding accuracy. These results provide the first evidence that a
simple RL algorithm can train VLMs entirely in cheap synthetic worlds while
delivering measurable gains on real-image agentic, spatial-reasoning, and
web-navigation benchmarks.
A lightweight, hyperparameter-free RL algorithm, VL-DAC, enables VLMs to learn generalized policies from inexpensive simulators, improving performance on real-world benchmarks without sacrificing image understanding accuracy.
Vision-Language Decoupled Actor-Critic (VL-DAC) PPO updates action tokens value learning MiniWorld Gym-Cards ALFWorld WebShop BALROG VSI-Bench VisualWebBench general image understanding accuracy