UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning
Graphical User Interface (GUI) agents have demonstrated remarkable progress
in automating complex user interface interactions through reinforcement
learning. However, current approaches face a fundamental dilemma: offline RL
enables stable training on pre-collected trajectories, but struggles with
multi-step task execution for lack of trajectory-level reward signals; online
RL captures these signals through environment interaction, but suffers from
sparse rewards and prohibitive deployment costs. To address it, we present
Semi-online Reinforcement Learning, a novel paradigm that simulates online RL
on offline trajectories. During each rollout process, we preserve the original
model output within the multi-turn dialogue, where a Patch Module adaptively
recovers the divergence between rollout and expert trajectories. To capture
long-term training signals, Semi-online RL introduces discounted future returns
into the reward computation and optimizes the policy with weighted step-level
and episode-level advantages. We further introduce Semi-Online Performance
(SOP), a metric that aligns better with true online performance, serving as a
practical and effective proxy for real-world evaluation. Experiments show that
ours Semi-online RL achieves SOTA performance among 7B models across four
dynamic benchmarks, with significant gains over the base model (e.g., +12.0% on
AndroidWorld, +23.8% on AITW), demonstrating significant progress in bridging
the gap between offline training efficiency and online multi-turn reasoning.
The code is available at https://github.com/X-PLUG/MobileAgent/tree/main/UI-S1.
68c8c9b3733e345e52ac1e66
Zhengxi Lu
claimed_verified
2025-09-16T09:42:11.100Z
676127cf11b19ea602bb202a
/avatars/dfd802a24bd63e509728159ebb1769f6.svg
Zhengxi Lu
user
LZXzju
68c8c9b3733e345e52ac1e67
Jiabo Ye
claimed_verified
2025-09-16T09:42:37.393Z
63cd1e04ff7cd335f0ddfa66
/avatars/8cca4ed96c699f53d4daabff0f6d6b56.svg
Jiabo Ye
user
Mizukiluke
68c8c9b3733e345e52ac1e68
Fei Tang
68c8c9b3733e345e52ac1e69
Yongliang Shen
claimed_verified
2025-09-17T13:06:06.785Z
5e1058e9fcf41d740b69966d
/avatars/ce74839ba871f2b54313a670a233ba82.svg
Yongliang Shen
user
tricktreat
68c8c9b3733e345e52ac1e6a
Haiyang Xu
claimed_verified
2025-09-16T09:42:34.060Z
645b10e80c73ea27d13f7aca
/avatars/95e565306472a15067440b5b43e07a6f.svg
xuhaiyang
user
xhyandwyy
68c8c9b3733e345e52ac1e6b
Ziwei Zheng
68c8c9b3733e345e52ac1e6c
Weiming Lu
68c8c9b3733e345e52ac1e6d
Ming Yan
68c8c9b3733e345e52ac1e6e
Fei Huang
68c8c9b3733e345e52ac1e6f
Jun Xiao
68c8c9b3733e345e52ac1e70
Yueting Zhuang
UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning
Graphical User Interface (GUI) agents have demonstrated remarkable progress
in automating complex user interface interactions through reinforcement
learning. However, current approaches face a fundamental dilemma: offline RL
enables stable training on pre-collected trajectories, but struggles with
multi-step task execution for lack of trajectory-level reward signals; online
RL captures these signals through environment interaction, but suffers from
sparse rewards and prohibitive deployment costs. To address it, we present
Semi-online Reinforcement Learning, a novel paradigm that simulates online RL
on offline trajectories. During each rollout process, we preserve the original
model output within the multi-turn dialogue, where a Patch Module adaptively
recovers the divergence between rollout and expert trajectories. To capture
long-term training signals, Semi-online RL introduces discounted future returns
into the reward computation and optimizes the policy with weighted step-level
and episode-level advantages. We further introduce Semi-Online Performance
(SOP), a metric that aligns better with true online performance, serving as a
practical and effective proxy for real-world evaluation. Experiments show that
ours Semi-online RL achieves SOTA performance among 7B models across four
dynamic benchmarks, with significant gains over the base model (e.g., +12.0% on
AndroidWorld, +23.8% on AITW), demonstrating significant progress in bridging
the gap between offline training efficiency and online multi-turn reasoning.
The code is available at https://github.com/X-PLUG/MobileAgent/tree/main/UI-S1.
Semi-online Reinforcement Learning addresses the limitations of offline and online RL by simulating online RL on offline trajectories, achieving state-of-the-art performance in dynamic benchmarks.
reinforcement learning offline RL online RL multi-step task execution trajectory-level reward signals sparse rewards deployment costs Patch Module discounted future returns step-level advantages episode-level advantages Semi-Online Performance (SOP) dynamic benchmarks AndroidWorld AITW