Morae: Proactively Pausing UI Agents for User Choices
User interface (UI) agents promise to make inaccessible or complex UIs easier
to access for blind and low-vision (BLV) users. However, current UI agents
typically perform tasks end-to-end without involving users in critical choices
or making them aware of important contextual information, thus reducing user
agency. For example, in our field study, a BLV participant asked to buy the
cheapest available sparkling water, and the agent automatically chose one from
several equally priced options, without mentioning alternative products with
different flavors or better ratings. To address this problem, we introduce
Morae, a UI agent that automatically identifies decision points during task
execution and pauses so that users can make choices. Morae uses large
multimodal models to interpret user queries alongside UI code and screenshots,
and prompt users for clarification when there is a choice to be made. In a
study over real-world web tasks with BLV participants, Morae helped users
complete more tasks and select options that better matched their preferences,
as compared to baseline agents, including OpenAI Operator. More broadly, this
work exemplifies a mixed-initiative approach in which users benefit from the
automation of UI agents while being able to express their preferences.
68b50fd6851c6e7b001eca48
Yi-Hao Peng
claimed_verified
2025-09-01T13:51:08.657Z
63edc468679c2cc40abd4b16
https://cdn-avatars.huggingface.co/v1/production/uploads/63edc468679c2cc40abd4b16/G6dSJ_tFG28QIRt9MaxHv.jpeg
Yi-Hao
user
yihaopeng
68b50fd6851c6e7b001eca49
Dingzeyu Li
68b50fd6851c6e7b001eca4a
Jeffrey P. Bigham
68b50fd6851c6e7b001eca4b
Amy Pavel
Morae: Proactively Pausing UI Agents for User Choices
User interface (UI) agents promise to make inaccessible or complex UIs easier
to access for blind and low-vision (BLV) users. However, current UI agents
typically perform tasks end-to-end without involving users in critical choices
or making them aware of important contextual information, thus reducing user
agency. For example, in our field study, a BLV participant asked to buy the
cheapest available sparkling water, and the agent automatically chose one from
several equally priced options, without mentioning alternative products with
different flavors or better ratings. To address this problem, we introduce
Morae, a UI agent that automatically identifies decision points during task
execution and pauses so that users can make choices. Morae uses large
multimodal models to interpret user queries alongside UI code and screenshots,
and prompt users for clarification when there is a choice to be made. In a
study over real-world web tasks with BLV participants, Morae helped users
complete more tasks and select options that better matched their preferences,
as compared to baseline agents, including OpenAI Operator. More broadly, this
work exemplifies a mixed-initiative approach in which users benefit from the
automation of UI agents while being able to express their preferences.
Morae, a UI agent, enhances accessibility for BLV users by involving them in decision-making processes during task execution, using large multimodal models to interpret user queries and UI elements.
UI agents blind and low-vision (BLV) users decision points large multimodal models user queries UI code screenshots mixed-initiative approach