VEM: Environment-Free Exploration for Training GUI Agent with Value
  Environment Model
Training Vision-Language Models (VLMs) for Graphical User Interfaces (GUI)
agents via Reinforcement Learning (RL) faces critical challenges:
environment-based RL requires costly interactions, while environment-free
methods struggle with distribution shift and reward generalization. We propose
an environment-free RL framework that decouples value estimation from policy
optimization by leveraging a pretrained Value Environment Model (VEM). VEM
predicts state-action values directly from offline data, distilling human-like
priors about GUI interaction outcomes without requiring next-state prediction
or environmental feedback. This avoids compounding errors and enhances
resilience to UI changes by focusing on semantic reasoning (e.g., Does this
action advance the user's goal?). The framework operates in two stages: (1)
pretraining VEM to estimate long-term action utilities and (2) guiding policy
exploration with frozen VEM signals, enabling layout-agnostic GUI automation.
Evaluated on Android-in-the-Wild benchmarks, VEM achieves state-of-the-art
performance in both offline and online settings, outperforming environment-free
baselines significantly and matching environment-based approaches without
interaction costs. Importantly, VEM demonstrates that semantic-aware value
estimation can achieve comparable performance with online-trained methods.
67bfd5d2381f8fcb67e5ad36
Jiani Zheng
claimed_verified
2025-03-02T20:18:18.436Z
64531f631a57e1179c203e6b
https://cdn-avatars.huggingface.co/v1/production/uploads/64531f631a57e1179c203e6b/C_J7pXFLqoJoHYPPhK3J9.jpeg
zjn
user
garlicisnotmyfavor
67bfd5d2381f8fcb67e5ad37
Lu Wang
67bfd5d2381f8fcb67e5ad38
Fangkai Yang
claimed_verified
2025-02-27T09:14:57.452Z
669dcf6200970c3b27aafa5d
/avatars/bb9ed5ff86326fdaeb184c6b0e40f74f.svg
kaikai yang
user
keanudicap
67bfd5d2381f8fcb67e5ad39
Chaoyun Zhang
claimed_verified
2025-02-27T09:14:59.653Z
654dbac9938fbf1e696be8aa
/avatars/b3c4035c48169c1bfb04a439fce3499f.svg
Chaoyun Zhang
user
vyokky
67bfd5d2381f8fcb67e5ad3a
Lingrui Mei
claimed_verified
2025-06-25T20:59:43.759Z
63120517ae8896941da4c5da
/avatars/10e1be026035f3e24225e6782a710083.svg
Lingrui Mei
user
Chevalier
67bfd5d2381f8fcb67e5ad3b
Wenjie Yin
67bfd5d2381f8fcb67e5ad3c
Qingwei Lin
67bfd5d2381f8fcb67e5ad3d
Dongmei Zhang
67bfd5d2381f8fcb67e5ad3e
Saravan Rajmohan
67bfd5d2381f8fcb67e5ad3f
Qi Zhang
VEM: Environment-Free Exploration for Training GUI Agent with Value
  Environment Model
Training Vision-Language Models (VLMs) for Graphical User Interfaces (GUI)
agents via Reinforcement Learning (RL) faces critical challenges:
environment-based RL requires costly interactions, while environment-free
methods struggle with distribution shift and reward generalization. We propose
an environment-free RL framework that decouples value estimation from policy
optimization by leveraging a pretrained Value Environment Model (VEM). VEM
predicts state-action values directly from offline data, distilling human-like
priors about GUI interaction outcomes without requiring next-state prediction
or environmental feedback. This avoids compounding errors and enhances
resilience to UI changes by focusing on semantic reasoning (e.g., Does this
action advance the user's goal?). The framework operates in two stages: (1)
pretraining VEM to estimate long-term action utilities and (2) guiding policy
exploration with frozen VEM signals, enabling layout-agnostic GUI automation.
Evaluated on Android-in-the-Wild benchmarks, VEM achieves state-of-the-art
performance in both offline and online settings, outperforming environment-free
baselines significantly and matching environment-based approaches without
interaction costs. Importantly, VEM demonstrates that semantic-aware value
estimation can achieve comparable performance with online-trained methods.
The proposed environment-free RL framework uses a pretrained Value Environment Model (VEM) for offline value estimation and policy optimization, achieving high performance in GUI automation with minimal interaction costs.
Reinforcement Learning Value Environment Model VEM semantic reasoning offline data GUI automation Android-in-the-Wild benchmarks