Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents
Developing autonomous agents that effectively interact with Graphic User
Interfaces (GUIs) remains a challenging open problem, especially for small
on-device models. In this paper, we present Ferret-UI Lite, a compact,
end-to-end GUI agent that operates across diverse platforms, including mobile,
web, and desktop. Utilizing techniques optimized for developing small models,
we build our 3B Ferret-UI Lite agent through curating a diverse GUI data
mixture from real and synthetic sources, strengthening inference-time
performance through chain-of-thought reasoning and visual tool-use, and
reinforcement learning with designed rewards. Ferret-UI Lite achieves
competitive performance with other small-scale GUI agents. In GUI grounding,
Ferret-UI Lite attains scores of 91.6%, 53.3%, and 61.2% on the
ScreenSpot-V2, ScreenSpot-Pro, and OSWorld-G benchmarks, respectively. For GUI
navigation, Ferret-UI Lite achieves success rates of 28.0% on AndroidWorld
and 19.8% on OSWorld. We share our methods and lessons learned from
developing compact, on-device GUI agents.
68dc8f6f4159d1f2418f9a95
Zhen Yang
68dc8f6f4159d1f2418f9a96
Zi-Yi Dou
68dc8f6f4159d1f2418f9a97
Di Feng
68dc8f6f4159d1f2418f9a98
Forrest Huang
68dc8f6f4159d1f2418f9a99
Anh Nguyen
claimed_verified
2025-10-22T15:47:08.706Z
61ee4dc39539bc0a3bbc8a68
/avatars/723f89e9c7afecdbd2854544bd7eb083.svg
Anh Tuan Nguyen
user
tuananh7198
68dc8f6f4159d1f2418f9a9a
Keen You
68dc8f6f4159d1f2418f9a9b
Omar Attia
68dc8f6f4159d1f2418f9a9c
Yuhao Yang
68dc8f6f4159d1f2418f9a9d
Michael Feng
68dc8f6f4159d1f2418f9a9e
Haotian Zhang
68dc8f6f4159d1f2418f9a9f
Ram Ramrakhya
68dc8f6f4159d1f2418f9aa0
Chao Jia
68dc8f6f4159d1f2418f9aa1
Jeffrey Nichols
68dc8f6f4159d1f2418f9aa2
Alexander Toshev
68dc8f6f4159d1f2418f9aa3
Yinfei Yang
68dc8f6f4159d1f2418f9aa4
Zhe Gan
Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents
Developing autonomous agents that effectively interact with Graphic User
Interfaces (GUIs) remains a challenging open problem, especially for small
on-device models. In this paper, we present Ferret-UI Lite, a compact,
end-to-end GUI agent that operates across diverse platforms, including mobile,
web, and desktop. Utilizing techniques optimized for developing small models,
we build our 3B Ferret-UI Lite agent through curating a diverse GUI data
mixture from real and synthetic sources, strengthening inference-time
performance through chain-of-thought reasoning and visual tool-use, and
reinforcement learning with designed rewards. Ferret-UI Lite achieves
competitive performance with other small-scale GUI agents. In GUI grounding,
Ferret-UI Lite attains scores of 91.6%, 53.3%, and 61.2% on the
ScreenSpot-V2, ScreenSpot-Pro, and OSWorld-G benchmarks, respectively. For GUI
navigation, Ferret-UI Lite achieves success rates of 28.0% on AndroidWorld
and 19.8% on OSWorld. We share our methods and lessons learned from
developing compact, on-device GUI agents.
Ferret-UI Lite, a compact end-to-end GUI agent, achieves competitive performance across diverse platforms using chain-of-thought reasoning, visual tool-use, and reinforcement learning.
chain-of-thought reasoning visual tool-use reinforcement learning GUI agent GUI grounding GUI navigation ScreenSpot-V2 ScreenSpot-Pro OSWorld-G AndroidWorld OSWorld