Test-Time Reinforcement Learning for GUI Grounding via Region
  Consistency
Graphical User Interface (GUI) grounding, the task of mapping natural
language instructions to precise screen coordinates, is fundamental to
autonomous GUI agents. While existing methods achieve strong performance
through extensive supervised training or reinforcement learning with labeled
rewards, they remain constrained by the cost and availability of pixel-level
annotations. We observe that when models generate multiple predictions for the
same GUI element, the spatial overlap patterns reveal implicit confidence
signals that can guide more accurate localization. Leveraging this insight, we
propose GUI-RC (Region Consistency), a test-time scaling method that constructs
spatial voting grids from multiple sampled predictions to identify consensus
regions where models show highest agreement. Without any training, GUI-RC
improves accuracy by 2-3% across various architectures on ScreenSpot
benchmarks. We further introduce GUI-RCPO (Region Consistency Policy
Optimization), which transforms these consistency patterns into rewards for
test-time reinforcement learning. By computing how well each prediction aligns
with the collective consensus, GUI-RCPO enables models to iteratively refine
their outputs on unlabeled data during inference. Extensive experiments
demonstrate the generality of our approach: GUI-RC boosts
Qwen2.5-VL-3B-Instruct from 80.11% to 83.57% on ScreenSpot-v2, while GUI-RCPO
further improves it to 85.14% through self-supervised optimization. Our
approach reveals the untapped potential of test-time scaling and test-time
reinforcement learning for GUI grounding, offering a promising path toward more
robust and data-efficient GUI agents.
6895643848b0ae5ca2710d5a
Yong Du
claimed_verified
2025-08-13T07:27:47.987Z
67ef9bdeeaf849151c740adf
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/-n7Ivd3RJnKZvFjF-z0ji.jpeg
Yong Du
user
DIONG1024
6895643848b0ae5ca2710d5b
Yuchen Yan
claimed_verified
2025-08-11T06:49:57.884Z
64098738342c26884c792c93
https://cdn-avatars.huggingface.co/v1/production/uploads/64098738342c26884c792c93/SxBUd-wLrl-PjQsrVYJte.jpeg
Yuchen Yan
user
yanyc
6895643848b0ae5ca2710d5c
Fei Tang
6895643848b0ae5ca2710d5d
Zhengxi Lu
claimed_verified
2025-08-22T07:24:54.864Z
676127cf11b19ea602bb202a
/avatars/dfd802a24bd63e509728159ebb1769f6.svg
Zhengxi Lu
user
LZXzju
6895643848b0ae5ca2710d5e
Chang Zong
6895643848b0ae5ca2710d5f
Weiming Lu
6895643848b0ae5ca2710d60
Shengpei Jiang
6895643848b0ae5ca2710d61
Yongliang Shen
claimed_verified
2025-08-11T06:50:01.160Z
5e1058e9fcf41d740b69966d
/avatars/ce74839ba871f2b54313a670a233ba82.svg
Yongliang Shen
user
tricktreat
Test-Time Reinforcement Learning for GUI Grounding via Region
  Consistency
Graphical User Interface (GUI) grounding, the task of mapping natural
language instructions to precise screen coordinates, is fundamental to
autonomous GUI agents. While existing methods achieve strong performance
through extensive supervised training or reinforcement learning with labeled
rewards, they remain constrained by the cost and availability of pixel-level
annotations. We observe that when models generate multiple predictions for the
same GUI element, the spatial overlap patterns reveal implicit confidence
signals that can guide more accurate localization. Leveraging this insight, we
propose GUI-RC (Region Consistency), a test-time scaling method that constructs
spatial voting grids from multiple sampled predictions to identify consensus
regions where models show highest agreement. Without any training, GUI-RC
improves accuracy by 2-3% across various architectures on ScreenSpot
benchmarks. We further introduce GUI-RCPO (Region Consistency Policy
Optimization), which transforms these consistency patterns into rewards for
test-time reinforcement learning. By computing how well each prediction aligns
with the collective consensus, GUI-RCPO enables models to iteratively refine
their outputs on unlabeled data during inference. Extensive experiments
demonstrate the generality of our approach: GUI-RC boosts
Qwen2.5-VL-3B-Instruct from 80.11% to 83.57% on ScreenSpot-v2, while GUI-RCPO
further improves it to 85.14% through self-supervised optimization. Our
approach reveals the untapped potential of test-time scaling and test-time
reinforcement learning for GUI grounding, offering a promising path toward more
robust and data-efficient GUI agents.
GUI-RC and GUI-RCPO enhance GUI grounding accuracy by leveraging spatial consistency and reinforcement learning without additional training data.
GUI grounding spatial overlap patterns spatial voting grids consensus regions test-time scaling test-time reinforcement learning ScreenSpot benchmarks Qwen2.5-VL-3B-Instruct ScreenSpot-v2