UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and
  Precise Inference-Time Grounding
The emergence of Multimodal Large Language Models (MLLMs) has driven
significant advances in Graphical User Interface (GUI) agent capabilities.
Nevertheless, existing GUI agent training and inference techniques still suffer
from a dilemma for reasoning designs, ineffective reward, and visual noise. To
address these issues, we introduce UI-AGILE, a comprehensive framework
enhancing GUI agents at both the training and inference stages. For training,
we propose a suite of improvements to the Supervised Fine-Tuning (SFT) process:
1) a Continuous Reward function to incentivize high-precision grounding; 2) a
"Simple Thinking" reward to balance planning with speed and grounding accuracy;
and 3) a Cropping-based Resampling strategy to mitigate the sparse reward
problem and improve learning on complex tasks. For inference, we present
Decomposed Grounding with Selection, a novel method that dramatically improves
grounding accuracy on high-resolution displays by breaking the image into
smaller, manageable parts. Experiments show that UI-AGILE achieves the
state-of-the-art performance on two benchmarks ScreenSpot-Pro and
ScreenSpot-v2. For instance, using both our proposed training and inference
enhancement methods brings 23% grounding accuracy improvement over the best
baseline on ScreenSpot-Pro.
6895adf548b0ae5ca2710e71
Shuquan Lian
claimed_verified
2025-08-08T16:19:02.176Z
669f53549a21428ccda89fab
/avatars/957c8251615ab4552f2e286ef7445c58.svg
LianShuQuan
user
LianShuQuan
6895adf548b0ae5ca2710e72
Yuhang Wu
6895adf548b0ae5ca2710e73
Jia Ma
6895adf548b0ae5ca2710e74
Zihan Song
6895adf548b0ae5ca2710e75
Bingqi Chen
6895adf548b0ae5ca2710e76
Xiawu Zheng
6895adf548b0ae5ca2710e77
Hui Li
UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and
  Precise Inference-Time Grounding
The emergence of Multimodal Large Language Models (MLLMs) has driven
significant advances in Graphical User Interface (GUI) agent capabilities.
Nevertheless, existing GUI agent training and inference techniques still suffer
from a dilemma for reasoning designs, ineffective reward, and visual noise. To
address these issues, we introduce UI-AGILE, a comprehensive framework
enhancing GUI agents at both the training and inference stages. For training,
we propose a suite of improvements to the Supervised Fine-Tuning (SFT) process:
1) a Continuous Reward function to incentivize high-precision grounding; 2) a
"Simple Thinking" reward to balance planning with speed and grounding accuracy;
and 3) a Cropping-based Resampling strategy to mitigate the sparse reward
problem and improve learning on complex tasks. For inference, we present
Decomposed Grounding with Selection, a novel method that dramatically improves
grounding accuracy on high-resolution displays by breaking the image into
smaller, manageable parts. Experiments show that UI-AGILE achieves the
state-of-the-art performance on two benchmarks ScreenSpot-Pro and
ScreenSpot-v2. For instance, using both our proposed training and inference
enhancement methods brings 23% grounding accuracy improvement over the best
baseline on ScreenSpot-Pro.
UI-AGILE enhances GUI agents through improved training with a Continuous Reward function, Simple Thinking reward, and Cropping-based Resampling, and inference with Decomposed Grounding with Selection, achieving state-of-the-art performance on GUI benchmarks.
Multimodal Large Language Models GUI agents Supervised Fine-Tuning Continuous Reward function Simple Thinking reward Cropping-based Resampling Decomposed Grounding with Selection ScreenSpot-Pro ScreenSpot-v2