BTL-UI: Blink-Think-Link Reasoning Model for GUI Agent
In the field of AI-driven human-GUI interaction automation, while rapid
advances in multimodal large language models and reinforcement fine-tuning
techniques have yielded remarkable progress, a fundamental challenge persists:
their interaction logic significantly deviates from natural human-GUI
communication patterns. To fill this gap, we propose "Blink-Think-Link" (BTL),
a brain-inspired framework for human-GUI interaction that mimics the human
cognitive process between users and graphical interfaces. The system decomposes
interactions into three biologically plausible phases: (1) Blink - rapid
detection and attention to relevant screen areas, analogous to saccadic eye
movements; (2) Think - higher-level reasoning and decision-making, mirroring
cognitive planning; and (3) Link - generation of executable commands for
precise motor control, emulating human action selection mechanisms.
Additionally, we introduce two key technical innovations for the BTL framework:
(1) Blink Data Generation - an automated annotation pipeline specifically
optimized for blink data, and (2) BTL Reward -- the first rule-based reward
mechanism that enables reinforcement learning driven by both process and
outcome. Building upon this framework, we develop a GUI agent model named
BTL-UI, which demonstrates consistent state-of-the-art performance across both
static GUI understanding and dynamic interaction tasks in comprehensive
benchmarks. These results provide conclusive empirical validation of the
framework's efficacy in developing advanced GUI Agents.
68d0ad048adc5cd018d15aa7
Shaojie Zhang
68d0ad048adc5cd018d15aa8
Ruoceng Zhang
68d0ad048adc5cd018d15aa9
Pei Fu
68d0ad048adc5cd018d15aaa
Shaokang Wang
68d0ad048adc5cd018d15aab
Jiahui Yang
68d0ad048adc5cd018d15aac
Xin Du
68d0ad048adc5cd018d15aad
Shiqi Cui
68d0ad048adc5cd018d15aae
Bin Qin
68d0ad048adc5cd018d15aaf
Ying Huang
68d0ad048adc5cd018d15ab0
Zhenbo Luo
68d0ad048adc5cd018d15ab1
Jian Luan
BTL-UI: Blink-Think-Link Reasoning Model for GUI Agent
In the field of AI-driven human-GUI interaction automation, while rapid
advances in multimodal large language models and reinforcement fine-tuning
techniques have yielded remarkable progress, a fundamental challenge persists:
their interaction logic significantly deviates from natural human-GUI
communication patterns. To fill this gap, we propose "Blink-Think-Link" (BTL),
a brain-inspired framework for human-GUI interaction that mimics the human
cognitive process between users and graphical interfaces. The system decomposes
interactions into three biologically plausible phases: (1) Blink - rapid
detection and attention to relevant screen areas, analogous to saccadic eye
movements; (2) Think - higher-level reasoning and decision-making, mirroring
cognitive planning; and (3) Link - generation of executable commands for
precise motor control, emulating human action selection mechanisms.
Additionally, we introduce two key technical innovations for the BTL framework:
(1) Blink Data Generation - an automated annotation pipeline specifically
optimized for blink data, and (2) BTL Reward -- the first rule-based reward
mechanism that enables reinforcement learning driven by both process and
outcome. Building upon this framework, we develop a GUI agent model named
BTL-UI, which demonstrates consistent state-of-the-art performance across both
static GUI understanding and dynamic interaction tasks in comprehensive
benchmarks. These results provide conclusive empirical validation of the
framework's efficacy in developing advanced GUI Agents.
A brain-inspired framework, Blink-Think-Link, enhances human-GUI interaction by mimicking cognitive processes and introduces innovations in data generation and reinforcement learning rewards.
multimodal large language models reinforcement fine-tuning brain-inspired framework Blink-Think-Link Blink Data Generation BTL Reward reinforcement learning GUI agent model BTL-UI static GUI understanding dynamic interaction tasks